[Feature Request] Define a sweep in config.yaml file rather than command line

[tomsercu]

🚀 Feature Request

Currently a sweep can only be defined on the command line. Could it be supported through the config files as well?

Motivation / pitch

Situations where this particularly makes sense: python train.py seed="range(0,10)" xval_fold="range(0,5)" actual_hyperparam_I_want_to_change=1,2,3
-- if for any new hyperparameter combination we always want to run over those several seeds and cross-validation folds.

For those cases I'd like to be able to have those sweeps in config.yaml, to declutter the command line.

Couldn't find prior discussions about this, but somewhat related is #394

cc @liujas000

[jieru-hu]

Defining a sweep in config file is supported in all Hydra sweepers(example) except for the BasicSweeper (the one you are using)

@omry does it make sense for us to add this support to the BasicSweeper?

[omry]

I think this can be added as a part of the change to standardize the configuration of the search space of HPO sweepers.

We have some work in that direction, but it's stalled. (cc @shagunsodhani).
I can't find a corresponding issue for this feature request.

To set expectations:
None of this is likely to happen in time for Hydra 1.1.

[zaccharieramzi]

I am very much interested in this feature, that's why I had opened a duplicate #1948 .

Reposting here an additional request that I had regarding this feature: do you think it would be possible to make it work with this type of sweep? i.e. python my_app.py -m '+top={a:10,b:20},{a:100,b:200}'

[Jasha10]

... do you think it would be possible to make it work with ... python my_app.py -m '+top={a:10,b:20},{a:100,b:200}'

Modifying the example from PR #1886, I've had success with the following config.yaml:

defaults:
  - db: ???
  - _self_

top: ???

hydra:
  sweeper:
    params:
      # db: glob(*)
      # db.timeout: 5,10
      top: "{a:10,b:20},{a:100,b:200}"

These settings for hydra.sweeper.params define a sweep that is equivalent to the following:

python my_app.py -m 'top={a:10,b:20},{a:100,b:200}'  # note: no plus symbol in front of "top"

Since PR #1948 does not implement the plus symbol '+' for adding to the config object, it's necessary to include a placeholder top: ??? in the user config.
Edit: It should be possible to use plus symbol as follows:

      +top: "{a:10,b:20},{a:100,b:200}"

[zaccharieramzi]

Amazing, sorry I didn't try to test it earlier.
Thanks for your reponsiveness.
I think this feature will greatly improve my setup.

[Jasha10]

I think this feature will greatly improve my setup.

Glad to hear it 😁

[zaccharieramzi]

@Jasha10 I have experimented a bit now, and I would like to know how you would go about trying to put the sweep part in a different configuration file (separate from the one defining the defaults and say the basic configuration parameters)?

I tried the following:

.
├── conf
│   ├── config.yaml
│   └── hydra
│       └── sweeper
│           └── params
│               └── grid_search.yaml
└── my_app.py

Where the content of grid_search.yaml is:

compile:
  lr: 1e-2,1e-3,1e-4

and the content of config.yaml is:

compile: ???

Running python my_app.py -m hydra.sweeper.params=grid_search, gives the following error:

Error merging override hydra.sweeper.params=grid_search
Cannot assign str to Dict[str, str]
    full_key: hydra.sweeper.params
    object_type=BasicSweeperConf

Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.

[Jasha10]

Thank you for testing the limits of this new feature, @zaccharieramzi.

First let me say that you will need python -m my_app.py hydra/sweeper/params=grid_search (with slashes) instead of python -m my_app.py hydra.sweeper.params=grid_search (with periods). I've just added an update to the docs with examples.

Here are two methods to achieve what you're going for, one of which depends on a PR that I have just opened and which is not yet merged into the main branch.

Method one: using an experiment directory:

This method works currently with the main branch of hydra.

$ tree
.
├── conf
│   ├── config.yaml
│   └── experiment
│       └── grid_search.yaml
└── my_app.py

2 directories, 3 files

conf/experiment/grid_search.yaml:

# @package _global_
hydra:
  sweeper:
    params:
      +compile.lr: 1e-2,1e-3,1e-4

This is using an @package directive.

Note that the plus symbol + preceding compiler.lr has the same meaning here as it does when using the CLI to modify the config object; using the syntax +compiler.lr: ... in yaml (as above) is equivalent to using the syntax +compiler.lr=... at the CLI.

At the command line:

$ python my_app.py -m +experiment=grid_search
[2022-01-12 20:23:38,705][HYDRA] Launching 3 jobs locally
[2022-01-12 20:23:38,705][HYDRA]        #0 : compile.lr=0.01 +experiment=grid_search
{'compile': {'lr': 0.01}}
[2022-01-12 20:23:38,790][HYDRA]        #1 : compile.lr=0.001 +experiment=grid_search
{'compile': {'lr': 0.001}}
[2022-01-12 20:23:38,882][HYDRA]        #2 : compile.lr=0.0001 +experiment=grid_search
{'compile': {'lr': 0.0001}}

Method two: selecting an option for 'hydra/sweeper/params'

This method is very close to what you tried above; it depends on PR #1956 and is not yet enabled on the main Hydra branch.

$ tree
.
├── conf
│   ├── config.yaml
│   └── hydra
│       └── sweeper
│           └── params
│               └── grid_search.yaml
└── my_app.py

conf/hydra/sweeper/params/grid_search.yaml:

+compile.lr: 1e-2,1e-3,1e-4

At the command line:

$ python my_app.py -m hydra/sweeper/params=grid_search hydra.job.chdir=true
[2022-01-12 20:34:33,921][HYDRA] Launching 3 jobs locally
[2022-01-12 20:34:33,921][HYDRA]        #0 : compile.lr=0.01
{'compile': {'lr': 0.01}}
[2022-01-12 20:34:34,006][HYDRA]        #1 : compile.lr=0.001
{'compile': {'lr': 0.001}}
[2022-01-12 20:34:34,101][HYDRA]        #2 : compile.lr=0.0001
{'compile': {'lr': 0.0001}}

[zaccharieramzi]

@Jasha10 Thanks for this reply!

I personally like the first option better, even with the potential copy-pasting I will have to do for different grid searches, because it allows me to have an experiment directory, which is very informative.

It could be nice to think about having a shortcut for:

# @package _global_
hydra:
  sweeper:
    params:

but certainly not for the next release.

I also tried the following syntax:

# @package _global_
hydra:
  sweeper:
    params:
      +compile: "{lr:1e-2,wd:1e-4},{lr:1e-3,wd:1e-5}"

and it worked perfectly as expected.

Can't wait for the next release, and actually I will probably use it right away ahah!

[GiorgioBarnabo]

Hi @Jasha10,

thank you for the explaination. Unfortunately, neither option is working in my case.

When trying method one, I get this error:

While for method two, I get:

Did something change? Maybe I am making mistakes in the config file? I kind of tried all possible combinations.

This is the python script with my_app():

Thank you in advance. Best,

Giorgio

[Jasha10]

@GiorgioBarnabo for method 1 to work, you will need to install a recent development version of Hydra (e.g. pip install --upgrade hydra-core==1.2.0.dev2) or install from the main branch here on github.

Method 2 is currently not working (as we have closed PR #1956).
Method 2 should work again once PR #2056 is finalized.