exp show: checkpoint experiment summary

[dberenbaum]

In dvc exp show, the last checkpoint for an experiment also contains the experiment name:

┏━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓
┃ Experiment              ┃ Created      ┃ State ┃ dvclive.json:step ┃ dvclive.json:acc ┃ mylive.json:step ┃ mylive.json:acc ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩
│ workspace               │ -            │ -     │ 3                 │ 0.9727           │ -                │ -               │
│ live                    │ Apr 19, 2021 │ -     │ -                 │ -                │ -                │ -               │
│ │ ╓ c1e9c94 [exp-f34fc] │ 10:45 AM     │ -     │ 3                 │ 0.9727           │ -                │ -               │
│ │ ╟ b97efe5             │ 10:45 AM     │ -     │ 2                 │ 0.9663           │ -                │ -               │
│ │ ╟ bc6accc             │ 10:45 AM     │ -     │ 1                 │ 0.9713           │ -                │ -               │
│ ├─╨ 796e9eb             │ 10:44 AM     │ -     │ 0                 │ 0.9538           │ -                │ -               │
│ │ ╓ 9197d13 [exp-c6a3d] │ 10:43 AM     │ -     │ 1                 │ 0.9133           │ -                │ -               │
│ ├─╨ 31df343             │ 10:43 AM     │ -     │ 0                 │ 0.8523           │ -                │ -               │
│ └── 7ada7e5             │ May 18, 2021 │ Queue │ -                 │ -                │ -                │ -               │
└─────────────────────────┴──────────────┴───────┴───────────────────┴──────────────────┴──────────────────┴─────────────────┘

It would be useful to have the experiment on a separate row to keep the experiment and individual checkpoints distinct.

The table combines info that's really about the experiment with info that's about the individual checkpoint. For example, parameters should be static for all checkpoints in an experiment, right?

Also, it opens up options to show things like summary metrics for an experiment, or to collapse the checkpoints if the user just wants a summary of each experiment.

[dberenbaum]

Related (#6174 (comment)):

   The Run status shows next to the already completed checkpoint.

This makes sense to me because when you are resuming/continuing a checkpoint run, you are starting from the last completed checkpoint (rather than starting from the queued row that only has params and no metrics/outputs)

To me it's like if I queue an experiment based on master, I wouldn't expect the master row to show Run when I run it.

[pmrowla]

For example, parameters should be static for all checkpoints in an experiment, right?

This is entirely dependent on what the user's stage code is doing. We just capture the state of the user's workspace, in theory the user can write to any file they want to before we capture that state, including params files.

And as we've discussed before, with the way the exp feature actually works, in theory users could leverage DVC to generate commits within an experiment ref that are not actually checkpoint experiments at all (i.e. just use them as a DVC-managed git branches). We would still want exp show to display whatever is in each of those commits, but could not make any assumptions about how they are actually supposed to be grouped or summarized

[dberenbaum]

This is entirely dependent on what the user's stage code is doing. We just capture the state of the user's workspace, in theory the user can write to any file they want to before we capture that state, including params files.

Won't any git-tracked or dvc-tracked changes generate a new experiment ref?

users could leverage DVC to generate commits within an experiment ref that are not actually checkpoint experiments at all (i.e. just use them as a DVC-managed git branches)

Good point, but is there any mechanism other than checkpoints where users could generate multiple commits to the same experiment? Obviously, it's possible users could manually mess with the experiments, but this seems like it would be an unexpected use.

[pmrowla]

Not currently, but there have been discussions about potentially allowing users to use the make_checkpoint signaling without checkpoint: true outs in their pipeline. So any stage could generate multiple commits in an experiment regardless of whether or not it was really a checkpoint experiment.

[dberenbaum]

For what purpose? It seems like holding dependencies fixed is the defining feature of an experiment. If parameters or dependencies change within an experiment, what is the meaning of an experiment? Would this ever be desirable?

[pmrowla]

Being able to track any kind of intermediate state during a stage or pipeline seems like it would be useful, it doesn't have to be restricted to intermediate checkpoints or circular deps.

[dberenbaum]

Even in checkpoints that are breaking down a multi-part stage without any kind of circular dependency, why would parameters change? Also, this scenario probably breaks all kinds of things in dvc exp show since it might not be an ML experiment with parameters or metrics at all.

A hyperparameter search script is one scenario where parameters might get changed within a stage, but my personal preference would be for each result to be tracked as a separate experiment, although I'm not sure how that would work. Otherwise, each checkpoint would include the history of all previous checkpoints even though they would be independent events in that scenario. Also, making them modified/independent experiments seems more consistent with current behavior where any parameter modifications generate a modified experiment.

[dberenbaum]

cc @daavoo