exp run: reset when the lock file is in Git

[dmpetrov]

Currently, the dvc exp run --reset option sets the checkpoints files to the latest commited dvc.lock (if that file is in Git). In many cases (probably in most of the cases), the training needs to be started from scratch but that seems not possible in the current design.

We should introduce something like --hard-reset to start training from scratch even for commited dvc.lock?

Also, I'm wondering about inconsistency between commited dvc.lock and not commited. It looks like --reset has a different behavior. Is there a way to unify it? The --hard-reser option does not unify it and I'm not sure it is the best way to solve the problem.

[dberenbaum]

Also, I'm wondering about inconsistency between commited dvc.lock and not commited. It looks like --reset has a different behavior. Is there a way to unify it? The --hard-reser option does not unify it and I'm not sure it is the best way to solve the problem.

Could you be more specific with this point? What is the inconsistency/different behavior you see?

[dmpetrov]

@dberenbaum from user's point of view, dvc exp run --reset has a different behavior when dvc.lock is commited and when it is not.

An example: if I need to start training from scratch is not enough to just run dvc exp run --reset. I need to check if lock file is commited or not.

After more thinking about this scenario... It feels like removing model and starting training from scratch is more frequent use case then starting from the previous commit. Instead of introducing --hard-reset we can change the default behavior to model removal while introduce --reset-to-previous for the case when the latest commited model is needed.

[dberenbaum]

It seems reasonable for the default --reset to always run the experiment from scratch (as if the stage has never been run before). One concern I would have is to make sure it doesn't just drop dvc.lock in case there are other pipelines that are not part of the experiment.

Rather than have a --reset-to-previous flag, maybe it makes more sense to have an option to drop all experiments in the workspace? Something like dvc exp remove --workspace. People may want to clear their workspace and start over, even in experiments without checkpoints. I have run into lots of times where I thought, "Whoops, didn't mean to do that, let me get back to a clean workspace." See also #5437.

Thoughts @pmrowla ?

[dmpetrov]

default --reset to always run the experiment from scratch

Yes, it seems like the best option.

Something like dvc exp remove --workspace

Is it equal to git checkout . && dvc checkout? If os, it is a good idea (this shortcut should be implemented I think) but it might not solve the issue because a user might need to train with previous model but the current code in the workspace.

It feels like a part of a bigger problem - how to granularly checkout a particular checkpoint and start training from it (without modifying workspace).

[dberenbaum]

Is it equal to git checkout . && dvc checkout? If os, it is a good idea (this shortcut should be implemented I think) but it might not solve the issue because a user might need to train with previous model but the current code in the workspace.

No, I mean dropping all experiments since the last commit. I probably shouldn't have called it the workspace (I was thinking of it as the opposite of dvc exp gc --workspace).

It feels like a part of a bigger problem - how to granularly checkout a particular checkpoint and start training from it (without modifying workspace).

Do you think this is common? I think it would be easy to implement but maybe confusing as a UI.

[dmpetrov]

how to granularly checkout a particular checkpoint and start training from it (without modifying workspace).

Do you think this is common? I think it would be easy to implement but maybe confusing as a UI.

Yes, I think so. For example, after a few code changes the model is still diverging. So, you decided to get the model from 50 epochs back but train on the latest code changes (in the current workspace). In this case, using dvc exp apply will destroy the workspace which you might not like. It can be something like dvc exp get HEAD^^ model.pth logs/

[pmrowla]

So to clarify action points here, the desired behavior would be:

• On dvc exp run --reset [target], any checkpoint outputs in target (or the entire pipeline if no target is provided) will be removed (regardless of whether or not dvc.lock contains an existing hash for that checkpoint out)
• Everything else in the workspace's dvc.lock will remain untouched

Rather than have a --reset-to-previous flag, maybe it makes more sense to have an option to drop all experiments in the workspace? Something like dvc exp remove --workspace. People may want to clear their workspace and start over, even in experiments without checkpoints. I have run into lots of times where I thought, "Whoops, didn't mean to do that, let me get back to a clean workspace." See also #5437.

I can see how this would be useful, but it's a bit of a separate issue from the checkpoint reset one.

Currently, if you have a "clean" workspace (with no existing experiments derived from HEAD), as long as HEAD contains a dvc.lock with a hash for some checkpoint output, that existing hash will be used as the starting point for that checkpoint out the next time you run exp run. In this scenario, it does not matter whether or not --reset is used, since there is no existing experiment to resume - any run would be a "new" checkpoint experiment.

However, with the proposed --reset change, the --reset flag would affect behavior in this scenario:

• without --reset, the new checkpoint experiment would use the checkpoint out hash from HEAD's dvc.lock as the starting point.
• with --reset, the new checkpoint experiment would start with a removed checkpoint out (ignoring the hash from dvc.lock

[dberenbaum]

Thanks, @pmrowla. That's a great summary. In the case of a clean workspace that contains a dvc.lock file, changing the behavior of --reset as you suggested makes sense to me (and makes the flag at least meaningful in this scenario). Any concerns?

[pmrowla]

This behavior will also make exp run more consistent with dvc repro, where a given run will respect any values present in dvc.lock at the start of the run (other than checkpoints with --reset). If a user wants a completely "clean" workspace before repro (and now exp run), they need to explicitly do it with git reset --hard.

[dberenbaum]

For example, after a few code changes the model is still diverging. So, you decided to get the model from 50 epochs back but train on the latest code changes (in the current workspace).

I believe that dvc exp run --rev would allow for this, right? Maybe I'm misunderstanding your point. As a different viewpoint, I'll also say that I'm unlikely to use this kind of workflow because changing code or params in the middle of an experiment kind of breaks clean experimentation.

This behavior will also make exp run more consistent with dvc repro, where a given run will respect any values present in dvc.lock at the start of the run (other than checkpoints with --reset). If a user wants a completely "clean" workspace before repro (and now exp run), they need to explicitly do it with git reset --hard.

I'm unclear on how exp run is more consistent with dvc repro than before. Can you explain? Are there cases outside of --reset where behavior has changed?

Getting back to a clean workspace requires removing all tracked and untracked files, so I think even git reset --hard may be insufficent if untracked files are present (git stash --include-untracked should work). Also, to clarify, this is not necessarily what I meant in the comment above. I meant something more like rm -rf .git/refs/exps but for only experiments based off HEAD. I can document that in a separate issue.