ENH support passing options through config files

[mathurinm]

proof of concept to start discussion.

Example of config file: ex.yml

objective-filter:
  - Lasso Regression[fit_intercept=True,reg=0.5]
dataset:
  - simulated
  - leukemia
solver:
  - celer
  - cd
n-repetitions: 1

used with:
benchopt run . --file ex.yml

These config files can be committed on repos for examples, shared across collaborators, are easier to maintain/modify/keep track of when the command to launch the benchmark is long

Questions:

• do we warn when a CLI option is replaced by an option in the config ?
• is there a way to avoid hardcoding the replacement of CLI option by specified config options ?
• discuss names in yml config vs parameters names in benchopt code
• support forced, pdb, html, do_profile, env_name in the config ?
• how to use autocompletion for this option?

[codecov-commenter]

Codecov Report

Merging #325 (8f31b06) into main (9013e53) will decrease coverage by 0.05%.
The diff coverage is 64.00%.

@@            Coverage Diff             @@
##             main     #325      +/-   ##
==========================================
- Coverage   55.58%   55.52%   -0.06%     
==========================================
  Files          42       42              
  Lines        2416     2433      +17     
  Branches      446      451       +5     
==========================================
+ Hits         1343     1351       +8     
- Misses        984      991       +7     
- Partials       89       91       +2     

[jolars]

Great idea! But isn't this syntax more kosher:

objectives:
  lasso:
    fit_intercept: true
    reg: 0.5
datasets:
  - simulated
  - leukemia
solvers:
  - celer
  - cd
repetitions: 1

or even:

objectives:
  - model: lasso
    fit_intercept: true
    reg: 0.5
  - model: slope
    fit_intercept: true
datasets:
  - simulated
  - leukemia
solvers:
  - celer
  - cd
repetitions: 1

do we warn when a CLI option is replaced by an option in the config ?

Shouldnt CLI options take precedence? Or are you talking about defaults?

[mathurinm]

The first syntax is a great idea and related to #193, it could be addressed when we progress on the latter.
For the 2nd, since benchmarks are in their own folder, I'm not sure config files should be shared between these folders.

[mathurinm]

I pushed a solution for the names differing in config file (where we want to use the name of the click options, e.g. --solver) and what is passed to the click decorated function (a variable solver_names).

To avoid hardcoding the dict, we could rely on click automapping from option to variable name (strip '--', replace '-' by '_') and do the reverse mapping automatically in _get_run_args.

WDYT @tomMoral @TomDLT

[mathurinm]

• Following @jolars suggestion I chose that CLI args override those specified in config_file. click knows when a parameter has its default value, so that's OK.

• I think in the config file parameters should be passed with their CLI option name (without the --) to ease documentation so it requires a bit of work to match the parameters from config_file with the ones from CLI, because click as already changed their names. Hence I have removed specific variable names in click.option (not an API breaking change), let click do the automatic conversion, and then assign variable names as the output of _get_run_args.

[TomDLT]

I agree with both points.

More generally, is there a good reason to have different names for CLI arguments and the function parameters ?

Support for pdb, env_name, etc. would be nice but is not necessary.
Same for autocompletion.

[mathurinm]

I'd like to add an example of config.yml file in the doc, on which page does it belong in your opinion? On the CLI page it doesn't look easy to place it properly (right after the run command?)

[jolars]

I'd like to add an example of config.yml file in the doc, on which page does it belong in your opinion? On the CLI page it doesn't look easy to place it properly (right after the run command?)

Right after the run command sounds fine to me.

[Klopfe]

I agree with Johan, I think after the run command is a good idea

[tomMoral]

This looks very neat! thanks @mathurinm

Could you add at least one test in test_cli.py? I think mimicking the TestRunCmd::test_benchopt_run passing only config should be enough.

[tomMoral]

LGTM!

there is just a small leftover --file in the test.

[tomMoral]

Thanks @mathurinm !!! :)