dvc exp run: revision for not queue/temp scenario

[dmpetrov]

DVC start each checkpoint from scratch when use --rev not from queue/temp:

$ dvc exp run --rev 452b20a --set-param lr=0.01
...

Accuracy is improving from 0.0 to ~0.16 in each checkpoint and starts from 0.0 again while it should start from 0.735 and got up with each of the checkpoint:

$ dvc exp show --no-pager
┏━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━┳━━━━━━━━┳━━━━━━━━━━┳━━━┳━━━━━━━━┳━━━┓
┃ Experiment              ┃ Created      ┃ step ┃   loss ┃ accuracy ┃ … ┃ epochs ┃ … ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━╇━━━━━━━━╇━━━━━━━━━━╇━━━╇━━━━━━━━╇━━━┩
│ workspace               │ -            │   41 │ 2.1555 │  0.16285 │ … │ 15     │ … │
│ master                  │ Mar 30, 2021 │    - │      - │        - │ - │ 15     │ … │
│ │ ╓ exp-09c97           │ 01:54 AM     │   41 │ 2.1555 │  0.16285 │ … │ 15     │ … │
│ │ ╟ 093d983             │ 01:54 AM     │   40 │ 2.1634 │   0.1629 │ … │ 15     │ … │
│ │ ╟ a8bb8f5             │ 01:54 AM     │   39 │ 2.1638 │  0.16078 │ … │ 15     │ … │
│ │ ╟ 85a61a9 (452b20a)   │ 01:54 AM     │   38 │ 2.2047 │  0.16487 │ … │ 15     │ … │
...

│ │ ╟ 452b20a             │ Apr 12, 2021 │   18 │ 1.4032 │    0.735 │ … │ 15     │ … │

PS: --temp option name is a bit confusing. I'd prefer something like --background.

[pmrowla]

So the issue here is that the intended workflow for workspace runs is that the user should do:

dvc exp apply 452b20a
dvc exp run --set-param lr=0.01

Right now what happens internally when with dvc exp run --rev 452b20a --set-param lr=0.01 is that an experiment is started based on 452b20a, but the current workspace state is also used on top of 452b20a. So the end result is that your current workspace's version of your model file is actually used in your new run, which is why you are seeing the wrong accuracy values in your new run.

--rev is really only meant to be used explicitly for --temp/--queue runs to account for the scenario where the user never applies anything to their workspace in the first place.

I think we should disallow --rev for workspace runs and enforce that the user do exp apply followed by exp run here. (enforcing this properly will also require the fix for #5737, since right now exp apply may fail due to the ambiguous commit issue)

@dmpetrov @dberenbaum thoughts?

[dmpetrov]

@pmrowla disabling should work. If it is a queue specific, does it make sense to extract it to dvc exp queue umbrella command? Might be related to #5615.

I see dvc exp run --rev 452b20a --set-param lr=0.01 as a really useful shortcut for apply+run for regular workflow (not queue). It would be great to have that feature. Does anything prevent us from generalization this feature for both of the scenarios?

CC @jorgeorpinel

[jorgeorpinel]

--rev is really only meant to be used explicitly for --temp/--queue runs

We currently say that "You can add a --rev to continue from a previous checkpoint" (workspace run context), and separately that --rev needed to resume --queued/temp runs. So are we saying that's incorrect and --rev is actually meant exclusively for out-of-workspace runs? Thanks

PS: --temp option name is a bit confusing. I'd prefer something like --background.

Extracted to #5826

[pmrowla]

@pmrowla disabling should work. If it is a queue specific, does it make sense to extract it to dvc exp queue umbrella command? Might be related to #5615.

@dmpetrov having the separate queue command is something that we will definitely need at some point, and it would make sense to move --rev into that set of commands at that time.

I see dvc exp run --rev 452b20a --set-param lr=0.01 as a really useful shortcut for apply+run for regular workflow (not queue). It would be great to have that feature. Does anything prevent us from generalization this feature for both of the scenarios?

We could implement it this way, but it's not clear what we should be doing with changes in the users current workspace. As a shortcut for a pure apply+run, we would remove any modifications that the user has in their workspace, which I'm not sure would be expected here (basically this would end up having the same effect as git reset --hard 452b20a at the start of the run)

We currently say that "You can add a --rev to continue from a previous checkpoint" (workspace run context), and separately that --rev needed to resume --queued/temp runs. So are we saying that's incorrect and --rev is actually meant exclusively for out-of-workspace runs? Thanks

@jorgeorpinel the current implementation does allow using --rev from in the workspace, so the current docs are technically correct. We're saying that we should change this and make it so --rev is only applicable for out of workspace runs.

[jorgeorpinel]

make it so --rev is only applicable for out of workspace runs

I see. So exp apply would be the way to "resume" from a certain checkpoint?

I guess that's what you said in #5814 (comment) (sry gets confusing with all the types of exps and runs).