ML experiments and hyperparameters tuning

[dmpetrov]

UPDATE: Skip to #2799 (comment) for a summary and updated requirements, and #2799 (comment) for the beginning of the implementation discussion.

Problem

There are a lot of discussions on how to manage ML experiments with DVC. Today's DVC design allows ML experiments through Git-based primitives such as commits and branches. This works nicely for large ML experiments when code writing and testing required. However, this model is too heavy for the hyperparameters tuning stage when the user makes dozens of small, one-line changes in config or code. Users don't want to have dozens of Git-commits or branches.

Requirements

A lightweight abstraction needs to be created in DVC to support hyperparameters-like tiny experiments without Git-commits. Hyperparameters tunning stage can be considered as a separate user activity outside of Git workflow. But the result of this activity still needs to be managed by Git preferably by a single commit.

High-level requirements to the hyperparameters tunning stage:

1. Run. Run dozens of experiments without committing any results into Git while keeping track of all the experiments. Each of the experiments includes a small config change or code change (usually, 1-2 lines).
2. Compare. A user should be able to compare two experiments: see diffs for code (and probably metrics)
3. Visualize. A user should be able to see all the experiments results: metrics that were generated. It might be some table with metrics or a graph. CSV table needs to be supported for custom visualization.
4. Propagate. Choose "the best" experiment (not necessarily the highest metrics) and propagate it to the workspace (bring all the config and code changes. Important: without retraining). Then it can be committed to Git. This is the final result of the current hyperparameter tunning stage. After that, the user can continue to work with a project in a regular Git workflow.
5. Store. Some (or all) of the experiments might be still useful (in additional to "the best" one). A user should be able to commit them to the Git as well. Preferably in a single commit to keep the Git history clean.
6. Clean. Not useful experiments should be removed with all the code and data artifacts that were created. A special subcommand of dvc gc might be needed.
7. [*] Parallel. In some cases, the experiments can be run in parallel which aligns with DVC parallel execution plans: Running DVC in production #2212, repro: add scheduler for parallelising execution jobs #755. This might not be implemented now (in the 1st version of this feature) but it is important to support parallel execution by this new lightweight abstraction.
8. Group. Iterations of hyperparameters tuning might be not related to each other and need to be managed and visualized separately. Experiments need to be grouped somehow.

What should NOT be covered by this feature?

This feature is NOT about the hyperparameter grid-search. In most cases, hyperparameters tuning is done by users manually using "smart" assumptions and hypotheses about hyperparameter space. Grid-search can be implemented on top of this feature/command using bash for example.

9. The ability to run the experiments from bash might be also a requirement for this feature request.

Possible implementations

This is an open question but many data scientists create directories for each of the experiments. In some cases, people create directories for a group of experiments and then experiments inside. We can use some of these ideas/practices to better align with users' experience and intuition.

Actions

This is a high-level feature request (epic). The requirements and an initial design need to be discussed and more feature requests need to be created. @iterative/engineering please share your feedback. Is something missing here?

EDITED:

Related issues

#2379
#2532
#1018 can be relevant (?)
Discussion

[casperdcl]

I think I almost-but-not-quite understand the aim here. I feel like I'm missing some key concept.

Run dozens of experiments without committing any results into Git while keeping track of all the experiments.

This seems to be almost a contradiction - the most robust way to "keep track" is to commit separately.

Each of the experiments includes a small config change or code change (usually, 1-2 lines).

This could be satisfied by for example by a bash script looping through param choices with @nteract/papermill for notebook users. I think it would be quite hard to try to write a tool to do this in a language/platform agnostic way. It's hard enough with papermill which is pretty niche.

To be all-encompassing we'd have to wind up supporting multiple ways of passing in params: env vars, cli args, sed -r 's/<search>/<repl>/g', and (nightmare) language-specific ways.

see diffs for code (and probably metrics)

Again a papermill-like approach (bash script spawning multiple notebooks and kernels, each with different params, each outputting a dvc metrics-like file) could do this

some table with metrics or a graph. CSV table needs to be supported for custom visualization.

Would need to create a formal metrics specification, or at least be very intelligent about automatically interpreting and visualising whatever the end-users throw at us.

Choose "the best" experiment (not necessarily the highest metrics) and propagate it to the workspace

Not sure how "best" can be automated with "not necessarily the highest metrics"

5. Store./6. Clean./7. [*] Parallel.

All could be handled by the bash script.

Experiments need to be grouped somehow.

Probably part of any potential formal metrics spec.

This feature is NOT about the hyperparameter grid-search

and

create directories for each of the experiments [...] directories for a group of experiments

Really seems like end-users writing bash/batch scripts would solve this.

---

Overall I feel like this has two requirements:

1. implement (or create) a formal metrics spec (which we can then use for visualisations etc)
2. document/add a tutorial for writing scripts to manage multiple experiments

I'd be against designing (1) from scratch owing to:

Also vaguely related maybe worth considering org-wide project boards (https://github.com/orgs/iterative/projects) for managing epics as well as cross-repo issues (e.g. treeverse/dvc.org#765 and iterative/example-versioning#5)

[dmpetrov]

@casperdcl good questions but let's start with the major one:

Run dozens of experiments without committing any results into Git while keeping track of all the experiments.

This seems to be almost a contradiction - the most robust way to "keep track" is to commit separately.

Let's imagine you are jumping to a hyperparameter tunning stage. You need to run a few experiments. You don't know in advance how many experiments are needed. Usually, it takes 10-20 but it might easily take 50-100.

Questions:

1. What abstraction would you choose? Commits to master? A new branch and commits in the branch? Is it okay for you to have 50 commits in a row?
2. You end up having 50 commits. How to get all the results and compare them to find the best result?
3. If Git abstractions work and new standards are not needed why a big portion of data scientists (including ex-developers) do not use this and prefer to create 50 dirs instead of 50 commits?

[casperdcl]

Run dozens of experiments without committing any results into Git while keeping track of all the experiments.

This seems to be almost a contradiction - the most robust way to "keep track" is to commit separately.

Ah I think we were both not using accurate language :) You do indeed want to commit results in some form (metrics for each experiment/summary of metrics/metadata to allow easy reproduction of experiments - which could just be the looping script). You don't necessarily want to commit runs (saved models, generated tweaked source code).

And when I said commit separately I should've just said commit. (Separately implies multiple commits, which isn't necessary unless you want to save each model and its outputs... which may actually still be useful. 1. Run multiple experiments 2. Save each in a separate branch commit 3. Collate metrics and use them to delete most branches. No clear advantage of this over multiple dirs. Maybe if you want to save the 2 best models on two different branches which will then fork?)

I think the rest of my comment dealt with the multi-dir, single-commit approach anyway (which as I understand is what you also intended).

[dmpetrov]

Yeah :) Sorry, I put the description in a very abstract form to not to push to any solutions. This abstract form gives a lot of opportunities for different interpretations which is probably the root cause of the misunderstanding. To be clear, I don't see any other solutions besides dirs yet but it would be great if we can consider other options.

I definitely want to give an ability to commit the results (both metrics as well as runs) but not necessarily all the results (ups to a user).

I think the rest of my comment dealt with the multi-dir, single-commit approach anyway (which as I understand is what you also intended).

👍

[Suor]

Preferably in a single commit to keep the Git history clean.

Doesn't sound like clean to me. It would be a very messy commit and if that experiments involve code changes it would be way easier to have a commit for each, this way you can git checkout it.

Additionally, if we have a git commit for each experiment we want to save then it would be very easy to save associated artifacts too.

Not useful experiments should be removed with all the code and data artifacts that were created

We might simply dvc run --no-commit, and no need to gc anything in the end.

Parallel. ... which aligns with DVC parallel execution plans:

Not necessarily, if we make a dir copy for each experiment, than that would be a different dvc repo, and we won't need any parallel processing for single repo.

[pared]

What worries me the most is the weight of the project. If we decide to go with dir approach, we need to either make a repo copy for each experiment or somehow link/use dvc controlled artifacts from original repo. I think that copy is fine for first version, but later we need to come up with something that would not duplicate our artifacts. That would probably align with parallel execution plans too.

[casperdcl]

About the whole dir copy thing... The bash loop + papermill workflow I gave as an example (granted only works for python notebooks) would create one dir per test, and said dir would only contain a notebook with one different parameter cell, as well as potentially some outputs. All notebooks would use (i.e. import) the same code and data from the root directory. And all you'd need to commit is the bash loop script & the metrics files from the output directories in order to reproduce/track what happened. May need random.seed(1337) or similar to reproduce identically but you get the idea.

My main concern is this all seems very language-, code layout-, and OS-specific and best left to the user to figure out. I think it would be helpful if we gave a concrete example of how dvc could assist in a workflow (e.g. this dummy C++ program training on this MNIST data on linux with a bash script subbing in (or passing in via CLI params) these 10 different params for 10 output dirs, running 2 jobs at a time, outputting metrics.csv, etc...)

I feel like trying to create an app to automate this process in generic scenarios is a bit like trying to create an app to help people use a computer. Sounds more like a manual/course than a product.

[dmpetrov]

Preferably in a single commit to keep the Git history clean.

Doesn't sound like clean to me. It would be a very messy commit and if that experiments involve code changes it would be way easier to have a commit for each, this way you can git checkout it.

Additionally, if we have a git commit for each experiment we want to save then it would be very easy to save associated artifacts too.

@Suor you are right, but ideally, it should be a user's choice - some folks are very against 50 commits and it would be great to provide some options to avoid this (if we can :) ).

In the dir-per-experiment paradigm, all the experiments might be easily saved in a single commit with all the artifacts (changed files and outputs) since they are separated. What do you think about this approach?

ADDED:

We might simply dvc run --no-commit, and no need to gc anything in the end.

Yeap. An additional, experiment specific option might be helpful like dvc repro --exp tune_lr

Parallel. ... which aligns with DVC parallel execution plans:

Not necessarily, if we make a dir copy for each experiment, than that would be a different dvc repo, and we won't need any parallel processing for single repo.

First, it looks like we have a bit different opinions regarding implementation. I assume that we copy all the artifacts in an experiment dir which gives us an ability to commit experiments (one by one or in a bulk). While you assume that we clone a repo to a dir. We can discuss the pros and cons of these methods. I won't be surprised if we find more options.

Thus, it depends on implementation. If it is a separate repo as a dir then we cannot commit it in the main repo. In this case, you are right in the above - separate commits above will be required.

If we run in a separate dir with no cloning (just copying and instantiating data artifacts) then we parallel running support might require.

[dmpetrov]

we need to either make a repo copy for each experiment or somehow link/use dvc controlled artifacts

@pared you are right. I don't think we can afford to make a copy of data artifacts. So, there is only one option - the most complicated one, unfortunately.

[dmpetrov]

My main concern is this all seems very language-, code layout-, and OS-specific and best left to the user to figure out.

Exactly. Notebook is kind of a specific language. I'd suggest building a language-agnostic version first based on config files or code file changes - copy all code in a dir, instantiate all the data files and run an experiment. Later we can introduce something more language/Notebook specific.

I think it would be helpful if we gave a concrete example of how dvc could assist in a workflow

Totally! We definitely need an example. This issue was created to initiate the discussion and collect the initial set of requirements. But the development process of MVP should be example-driven.

I feel like trying to create an app to automate this process in generic scenarios is a bit like trying to create an app to help people use a computer. Sounds more like a manual/course than a product.

I see this as an attempt to help users use one of the "best practices" - save all the experiments (in dirs :) ) and compare the results.

[jorgeorpinel]

What about trying to automatically generate a Git submodule for experiments? 1. Somehow mark code or data files as "under experimentation". 2. Watch those files and make a commit every time they're written (similar to IPython Notebook checkpoints) 3. Tell DVC to stop watching this experiment.

And do we have any ideas on what the interface would look like? Another command, a separate tool, a UI?

a big portion of data scientists (including ex-developers)... prefer to create 50 dirs instead of 50 commits

If this is the case. Perhaps a file linking system or UI that shows the user a growing set of virtual dirs simultaneously, one per experiment. Either based in the single-commit, multiple dir strategy, the git submodule, or something else.

[pared]

@dmpetrov Do you think that we could restrict (at least in the beginning) experiments feature to systems where linking is possible? That would eliminate the risk of experiment throttling disk space. Also in such case implementation does not seem too hard. We would just need to create a repo with default *-link cache type and point cache to the master project cache.

[dmpetrov]

What about trying to automatically generate a Git submodule for experiments?

@jorgeorpinel Might be I didn't get the idea but Git-submodule means a separate repo. So, we end up having 50 Git repositories instead of 50 commits. It looks like an even more havier approach that we currently have.

And do we have any ideas on what the interface would look like? Another command, a separate tool, a UI?

Initially, the command line one. I see that as part or repro. Line vi config.yaml && dvc repro --exp tune_lr - will create a dir with changed files and new outputs.