Introduce hyper parameters and config

[dmpetrov]

For an ML experiment, it is important to know metrics as well as the parameters that were used in order to get the metrics. Today there is no training/processing parameter concept in DVC which creates a problem when a user needs to visualize an experiment for example in some UI.

A common workaround is to track parameters as metrics. However, the meaning of metrics is different. All the UI tools (including dvc metrics diff) need to show deltas where deltas do not make sense to some types of params. For example, delta for learning rate might be ok to see (values are still better), but delta for a number of layers (32, 64 or 128) does not make sense, the same for not numeric params like strings.

Also, config/parameters are a pre-requisite for experiment management (#2799 or CI/CD scenarios) when DVC (or other automation tools) need to change training regarding provided parameters.

Another benefit of the "understanding" parameter - DVC can use this information during repro. For example, DVC can realize that a step process which depends on config file config.json should not be run despite the config file change because the metrics it uses were not changed.

We need to introduce the experiment config file/parameters file with a fixed structure that DVC can understand.

Open questions:

1. Filename. config.json, dvcconfig.json, params.json.
2. File format: json, text config/ini (name=value), Hydra, ... We can run a survey.
3. How to track param dependency for stages. We can introduce a new type of dependency: param. If it is given then the stage depends on the file and on particular params values. Like dvc run -p learning_rate -p LL5_levels ....
4. DVC should probably support groups of params. Param name pattern could be used : dvc run -p 'processing.*' ...

[elgehelge]

Great initiative @dmpetrov. What do you imagine the files contain? Is it stuff like required=true or is it types like learning_rate=float or is it default values like learning_rate=3.5e-5?

[Edit] Ah, so this is not about being able to run a pipeline with command line parameters? You are just talking about adding parameters as a concept, which would be just a file, right?

[elgehelge]

I think putting parameters in a file might be a step in the wrong direction, but I need to think a little more 🤔

[dmpetrov]

@elgehelge yes, it is about config with options that you can read from your code.

Having a config file that many ML pipeline stages use (and depend on) is a quite usual approach. We just need to specify a subset of params instead.

I'd love to hear other options!

[elgehelge]

Okay. A config/params file could be a good first step, but not having parameters as commandline arguments would make day-to-day development a bit cumbersome: edit file, run repro, edit file, run repro, and so on... , and also it would be hard to plug external tool for running jobs into this.

I do like the idea of having a file of default parameters though. But the important word here is default, since it would still be nice to be able to overwrite them using the commandline, but maybe this is another issue that comes later 😅

1. File name: Naming. config.json might conflict with other project related configs. And any name containing the word "config" makes me think that it contains settings. I am all for params.json.
2. File format: First I thought json. But then I saw that ini files have groups, which I find super useful. For instance, at a previous job we hardcoded 3 different sets of default parameters into the code, one for training a tiny model (for debugging purposes), one for training a model for a small segment of clients, and one for training a full model. It could also be that maybe for external compute or CI you would like to train for longer, so you could have a params specifically for this. And I guess there are plenty more use-cases where you might want more than one set of defaults. For this reason a like ini, however, two very strong reasons not to pick ini: (1) There is no formal standard, (2) There is no types, (3) Arrays er annoying to define. So, while researching on this I found TOML, which seems to solve all the problems related to ini (https://en.wikipedia.org/wiki/TOML)
3. Yes! So the dvc file could look something like this maybe??

md5: fd75c91bb60baf9bf36e4f64f0c1c7e0
cmd: python train_model.py
deps:
- md5: 53184cb9a5fcf12de2460a1545ea50d4
  path: train_model.py
- md5: 809776eaeef854443d39e4f76bd64c84.dir
  path: data/split
- json: '0.0005'
  param: learning_rate
outs:
- md5: 47b3781954c760e1bd19ac9e56e141b1
  path: model/mymodel.pkl

4. I love it.

Together with "the build cache" (#1234) I think this would work quite well.

[Edit] Removed me discussion about commandline support for parameters, let's take that somewhere else.

[elgehelge]

While taking a step in this direction, we might also want to consider to go all the way and add environment variables as a dependency. Maybe like so?

deps:
- env: TF_ADJUST_SATURATION_FUSED
  value: 1

It is not something I fin important myself, not might be worth considering.

[dmpetrov]

@elgehelge thank you for the feedback. All very helpful.

1. Filename. Agree that config is too broad. params or dvcparams are better options.
2. ini and json are definitely the most common cases. We actually can support both: params.conf or params.json (error if both are presented).

Note, for automation scenarios and visualization, we need a command to show all the params in a unified way like:

$ dvc params
learning_rate = 0.005
num_classes = 1000
augument.VerticalFlip = True
augument.Rotate = [20, 60, 90]
process.UseOpenCL = True

# The actual params file content:
$ cat params.conf
learning_rate = 0.005
num_classes = 1000

[augument]
VerticalFlip = True
Rotate = [20, 60, 90]

[process]
UseOpenCL = True

@elgehelge also, there is some misunderstanding in dvc run/repro. We need a way of defining params in dvc run and changing params in dvc repro.

Examples:

Action	Command
Define dependency to param in default params file params.conf	dvc run -p learning_rate -d train.py -d images/ -o model.pkl python train.py
Define dependency to a param in a custome params file my_param.conf	dvc run -P my_param.conf -p learning_rate -d train.py -d images/ -o model.pkl python train.py
Change a param in current workspace and repro	dvc repro model.pkl.dvc -p learning_rate 0.004
Change a param in branch, repro and commit	dvc repro model.pkl.dvc -p learning_rate 0.004 --branch new_lr --commit-message "try new learning rate 0.004"

Note, in both dvc repro examples the command depends on the param file. However, it should NOT retrain the model if the params file changed without the corresponded param change.

PS: sorry for being too detailed. I'm trying to provide as many details as possible to make sure everybody on the same page.

[elgehelge]

@dmpetrov Great examples. Two comments.

• Is there really a need for the "Change a param in branch, repro and commit"-action? Would it be nicer to just make a new branch, do dvc repro model.pkl.dvc -p learning_rate 0.004, and then commit? In a real world scenario I think the most common flow would be a bit different: Imagine that you just used the default learning rate, and now you want to tune it. First, you add support for a new parameter in your code, and then you add the parameter as a dependency. So this connot be done in a single command anyways.

• Maybe I was not being clear in my previous comment. I think it is important to consider the use-case of having multiple "parameters sets". One example could be this. Let's say you have a model that takes a whole day to train, but you would still like a CI to test that the code can run. It that case we would like a parameter set for a small fast model, and a parameter set for the real deployable one. Using toml this could look like:

[bigmodel]
learning_rate = 0.005
epochs = 25
data.Languages = ["english", "spanish", "chinese"]
data.MaxSamples = -1

[testmodel]
learning_rate = 0.005
epochs = 2
data.Languages = ["english"]
data.MaxSamples = 1000

And using json, you would need to have two different files, and thus two different stages. I am not saying one is better than the other, we just have to discuss it.

There is probably many other use-cases of having multiple parameter sets? Like having multiple customers which require different model settings (maybe the data needs different preprocessing).

[dmpetrov]

Would it be nicer to just make a new branch, do dvc repro model.pkl.dvc -p learning_rate 0.004, and then commit? .... First, you add support for a new parameter in your code, and then you add the parameter as a dependency. So, this cannot be done in a single command anyways.

@elgehelge, yes, this is the scenario "Change a param in current workspace and repro". You are right that this is a default mode for brand new parameters or functionality.

While the last scenario "Change a param in branch, repro and commit" is automation on top of the previous one. It is needed when a parameter was already extracted and you do hyper param search in an automated or semi-automated way - from a script for example or through some service.

I think it is important to consider the use-case of having multiple "parameters sets"

Very good point! It was not mentioned in the description. Happy to discuss this.

My thoughts... In the current proposal, the params could be easily overwritten dvc repro -p learning_rate 0.004. This is not enough for the use case you brought. To support your use case we can overwrite the entire param file like dvc repro --param-file params_test.conf -p learning_rate 0.004 while the default dvc repro will work with params_conf.conf which was defined in dvc run --param-file params_prod.conf or a default one params.conf.

@elgehelge what do you think about this proposal?

1. Is it ok to keep separate files, not separate sections? What is your preference?
2. Do we need a default params.conf or we should force users to specify it in each dvc run?
3. Any other ideas?