exp show: only shows one row while `exp run` is running.

[rogermparent]

Bug Report

Description

When running exp show between checkpoints of exp run (between checkpoints while the repo isn't locked), the table only shows a workspace and a single row (I assume the one currently being run)

In my case, this is the normal table before a run:

the table during a run:

and the table after the run is complete:

Reproduce

1. Set up dvc-checkpoints-mnist (or the demo project of the vscode extension)
2. Run dvc exp run
3. While dvc exp run is running, try running dvc exp show until the lock isn't present and you get a table.

Expected

The exp show table displays the full experiments table, or at least refuses to display the incorrect table.

Environment information

Output of dvc doctor:

DVC version: 2.0.10 (pip)
---------------------------------
Platform: Python 3.7.3 on Linux-5.8.0-3-amd64-x86_64-with-MX-19.3-patito_feo
Supports: http, https, s3
Cache types: hardlink, symlink
Cache directory: ext4 on /dev/nvme0n1p3
Caches: local
Remotes: https
Workspace directory: ext4 on /dev/nvme0n1p3
Repo: dvc (subdir), git

Additional Information (if any):

I'm aware this may be an unavoidable part of how experiments are implemented, and if so we could also use this Issue to discuss how to best work around it both as a user and as an external consumer of DVC CLI's output.

[pmrowla]

This is a side effect of how workspace experiment runs work. As a part of the various git operations that happen during an experiment run, HEAD gets moved around. At this particular point in time it will be a detached HEAD pointing to the current "checkpoint" commit.

By default, dvc exp show only shows experiments derived from (branched off of) the HEAD. So when HEAD is in this intermediate detached state, there are no additional experiment commits/branches that fit the criteria to be displayed.

So basically the issue is that exp show is not intended to be used in this way (during a run).

[pmrowla]

Internally we do know where the original HEAD points to (since we need to restore it after the run finishes). If exp show is used mid-run in this way, we can output the table with respect to that original HEAD.

I think this exp show change will be non-trivial though, making exp show work properly during execution will likely require adjusting how our repo lock works (related: #5739, #3783), and it seems like there's some potential for git-related race conditions that will need to be accounted for as well.

[shcheklein]

@pmrowla

Thanks a lot for explaining this 🙏 This issue (and the locks one) feel like very important ones from the VS code extension perspective.

This is a side effect of how workspace experiment runs work. As a part of the various git operations that happen during an experiment run, HEAD gets moved around. At this particular point in time it will be a detached HEAD pointing to the current "checkpoint" commit.

could you please elaborate a bit more on this? I guess describe (or point us) into how workspace experiment runs exactly
work. It's a bit hard for me to "connect the dots" for the description.

So basically the issue is that exp show is not intended to be used in this way (during a run).

is DVC supposed to be locked? how is it possible that we can dvc exp show at all in this case?

[pmrowla]

could you please elaborate a bit more on this? I guess describe (or point us) into how workspace experiment runs exactly
work. It's a bit hard for me to "connect the dots" for the description.

(checkpoint) experiment runs work the same as a typical DVC workflow like:

# HEAD points to some commit in `master`
$ dvc repro
$ git add -A
$ git commit
# HEAD now points to the new commit
$ dvc repro
$ git add -A
$ git commit
# HEAD now points to the new commit
...

Except that the commits we make are contained in experiment refs as opposed to a regular git branch (like master). While we are doing these git operations, git HEAD is still moved with each new experiment commit, in the same way that HEAD moves when you add a commit to a regular branch.

After the experiment run finishes, we restore HEAD to the user's initial commit.

Note that this is specific to workspace runs - in experiments, we also support the --queue/--temp runs where execution happens in a completely separate temporary directory environment, in this case the git HEAD for the users workspace would never change.

is DVC supposed to be locked? how is it possible that we can dvc exp show at all in this case?

The repo gets unlocked during repro/exp run when executing users stage commands, to allow for read-only commands to be run. This works fine for the usual dvc repro use case, but clearly needs to be revisited for experiments since we now modify the git repo state by moving HEAD. (exp show is the obvious example of why it's an issue, but this would also affect params/metrics diff commands that default to diffing against HEAD)

[shcheklein]

Thanks @pmrowla

After the experiment run finishes, we restore HEAD to the user's initial commit.

just clarify, does it mean that we also do dvc exp apply at the end?

making exp show work properly during execution will likely require adjusting how our repo lock works

what is your take on this? it seems that dvc exp show is read-only ... not clear immediately why do we need any changes to the existing locking mechanism to be honest in this case (clearly, we might need some additional locks to prevent Git race conditions, etc)

[pmrowla]

@shcheklein

just clarify, does it mean that we also do dvc exp apply at the end?

For workspace runs yes, the end result in the workspace will contain the original HEAD with the changes from the experiment run. Technically, we don't actually run apply, we just do git reset <ORIG_HEAD> (but the end result is equivalent to what the user would get with dvc exp apply).

For non-workspace runs (--temp or --queue+--run-all), the changes for each experiment are not applied to the workspace at all, and a user must manually run dvc exp apply if they actually want to see the changes in their workspace.

making exp show work properly during execution will likely require adjusting how our repo lock works

what is your take on this? it seems that dvc exp show is read-only ... not clear immediately why do we need any changes to the existing locking mechanism to be honest in this case (clearly, we might need some additional locks to prevent Git race conditions, etc)

The existing locking mechanism works, but as you noticed it doesn't handle potential git race conditions with regard to experiments, which is what I was getting at w/that comment.

[dberenbaum]

One other consideration mentioned by @pmrowla: similar issues may be found in the metrics and plots commands.

@rogermparent There's no timeline for this yet, and it might not be a quick fix, but it's on the dvc backlog, and you can track discussions here and any progress in the linked projects.

[jorgeorpinel]

As a part of the various git operations that happen during an experiment run, HEAD gets moved around.

Interesting. Another option may be to revert the behavior so that experiments are run as --tmp by default, and add a --debug option to run exps in the workspace. I think not being able to debug the pipeline was the main reason to make workspace runs the default, right? Not sure that would help for the VSCode extension though.

And if that's done, then I think it's more justifiable that exp show works differently during debug (a.k.a. workspace) runs; we can document it. It could even print a warning or error and/or be disabled in that scenario.