Checkpoint: separate model tuning and redesign behavior by a new command

[dmpetrov]

The current checkpoint behavior works nice in the model tuning stage but it is very painful to make code/model redesign changes and start training from scratch. It seems like we need to separate these types of users' activities by two different commands dvc exp run and dvc exp continue.

Another confusion with the current design - it is not correct to require checkpoints to be --always-changed. They should be reproducible by default. dvc exp run should start from scratch if no direct ask for --continue. Also, I’m not 100% sure that --persist is the right behavior. It feels like checkpoint is a special type of outputs.

How checkpoint reproducibility should look like:

$ dvc exp run # runs (let say) 27 epochs then Ctrl+C
$ dvc exp run # it should recognize checkpoints and say something like:
Stage 'train' didn't change. To continue training checkpoints use 'dvc exp continue' command.

What is the next after canceling:

1. Continue training with no changes. dvc exp continue - starts from the last step. It should add new checkpoints to the same experiment (the same level of hierarchy)
2. Change workspace. vi train.py & dvc exp run will start training from scratch (in a regular way with removing all outputs)
3. Change workspace and continue. vi train.py & dvc exp continue should fail and say that it is not possible since workspace was changed - please run another experiment dvc exp run
4. Manual checkout of a checkpoint. dvc exp checkout e4253be, changing code/params vi train.py and run it as a separate experiment: it can be a new training dvc exp run or continue with the latest checkpoint dvc exp continue. In any of these cases, it creates a new experiment (it would be amazing to show it as a new level in the hierarchy since it was inherited from another experiment - nice-to-have priority).
5. Automatic checkpoint change. The same as the previous one but automated with hyper params like dvc exp continue --params train.dropout 0.18 e4253be or starting from scratch dvc exp run --params train.dropout 0.18 e4253be

UPDATE 11/2 1am PST: I forgot to specify sha e4253be in the last scenario - updated.

Any feedback is highly appreciated. CC @iterative/engineering

[pmrowla]

It seems to me like we will end up needing to define a proper checkpoint output type after all, rather than using the persist + always_changed workaround to get the internal behavior we need.

Just to clarify the current behavior, regarding

1. Continue training with no changes. dvc exp continue - starts from the last step. It should add new checkpoints to the same experiment (the same level of hierarchy)
2. Change workspace. vi train.py & dvc exp run will start training from scratch (in a regular way with removing all outputs)

This is how dvc exp run --continue and dvc exp run --checkpoint are currently intended to work, but based on @dmpetrov's testing, there's some bugs in --continue and --reset behavior that I'm still investigating.

On the other points, we currently don't have an way of tracking which experiment is currently "checked out" in the user's workspace. Right now we just diff the workspace with the user's checked out git commit, and then compare that to see if there is a matching existing experiment. This works for regular experiments, but it looks like it won't be sufficient for checkpoints, especially if we want to support "branching" off of an existing checkpoint run.

Currently, if you want to get this branching behavior, you need to dvc exp checkout the specific checkpoint you want to use as a starting point, and promote that to a regular commit in your main/parent repo (git add + git commit) and then make your changes and start a new checkpoint run which is based on that new commit.

[Suor]

In computer vision it's common enough to train a new model starting from an old one, so always starting from scratch when code or data changes might be limiting to those guys.

[dmpetrov]

It seems to me like we will end up needing to define a proper checkpoint output type after all

Yes, it seems like it is needed.

Re the possible bug - yes, it seems like we discuss this but it not how the current version works.

Right now we just diff the workspace with the user's checked out git commit, and then

@pmrowla That's a good point. It means in the manual scenario (4) the next experiment cannot be inherited from the original one (without manually specified version). But auto scenario (5) should work, right? Are there any other issues related to this limitation?

Currently, if you want to get this branching behavior

Yeah, the requirement of creating a commit in the main repo is quite a limitation. Let's not focus on this branching behavior. As far as I understand, we still will be able to start from some checkpoints, but we are losing the connection between experiments. Is it correct?

[dmpetrov]

In computer vision it's common enough to train a new model starting from an old one, so always starting from scratch when code or data changes might be limiting to those guys.

@Suor This proposal is not about always starting from scratch. It is about explicitly supporting two modes - from scratch or from checkpoints. A scientist will be using mostly one of the modes depending on the stages he/she is in - development (when model changes and checkpoints are not compatible) or tuning (compatible checkpoints).

[pmrowla]

@pmrowla That's a good point. It means in the manual scenario (4) the next experiment cannot be inherited from the original one (without manually specified version). But auto scenario (5) should work, right? Are there any other issues related to this limitation?

Yeah, the requirement of creating a commit in the main repo is quite a limitation. Let's not focus on this branching behavior. As far as I understand, we still will be able to start from some checkpoints, but we are losing the connection between experiments. Is it correct?

Right, if we do something like continue abc123 --params ... then we could support branching directly from a checkpoint without dealing with the need to create a commit in the main repo. The main limitation for 4 & 5 is with editing params or code in the workspace, but specifying params via the command line would work for us.

[pmrowla]

Summary of discussion with @dmpetrov:

TODO:

• Use dedicated checkpoint output type rather than using persist outputs + --checkpoint run options (experiments: add checkpoint output type #4845)
• dvc exp run will behave the same way as current --checkpoint --reset without requiring extra checkpoint options (checkpoints: exp run and exp res[ume] refactor #4855)
  • If pipeline/stage contains checkpoint outputs, generate (or restart from scratch) a new checkpoint experiment branch
  • If pipeline/stage does not contain checkpoint outputs, generate a regular standalone experiment
• dvc exp resume will behave same way as current --checkpoint --continue (checkpoints: exp run and exp res[ume] refactor #4855)
• dvc exp resume should support branching from intermediate checkpoint w/params changes specified from command line (experiments: support branching w/modified params from intermediate checkpoint #4898)
  • Workspace based code changes will remain unsupported for now (keep current behavior requiring exp checkout <intermediate checkpoint> and promoting to a new commit first)
• set DVC_ROOT environment variable during stage execution to simplify checkpoint signaling for non-python user code (since signal file location depends on executor tempdir dvc root directory) (checkpoints: set DVC_ROOT environment variable #4877)