`exp show`: add output of `params diff` & `metrics diff` to `workspace` entry within `--show-json`

[mattseddon]

In the VS Code extension we recently added in a feature to our experiments table which highlights params and metrics which have changed with respect to the previous commit.

Example (example-dvc-experiments)

Given:

1. dvc params diff --show-json = {"params.yaml": {"train.epochs": {"old": 10, "new": 11, "diff": 1}}}
2. dvc metrics diff --show-json = {}

Then we will show:

The current cli implementation means that we now have to run 3 commands to get the required data for our table. These are:

1. dvc exp show --show-json
2. dvc params diff --show-json
3. dvc metrics diff --show-json

We are still running into repo locks when trying to run any of the above commands. Also, having to run all three is slower.

Ideally I'd like to include the output of 2 & 3 within the workspace dict of 1 but open to other suggestions.

Note: If we pushed this data into exp show then we would also be able to highlight the same records in the cli table.

Related issues:

• exp show: Allow running while DVC is locked #5739
• exp show: show only changed parameters #5966
• exp show: Include data files. #6434
• params/metrics/exp diff: consistent output #5515

[pmrowla]

You shouldn't need the extra params diff and metrics diff calls. The original commit data should already be available in exp show in <orig_commit_sha>.baseline.data

❯ dvc exp show --show-json
{
  "workspace": {
    "baseline": {
      "data": {
        ...
      }
    }
  },
  "00071e8d7f137fc7265b386aaba06d39a1a30b61": {
    "baseline": {
      "data": {
        "timestamp": "2021-05-29T15:23:32",
        "params": {
          "params.yaml": {
            "data": {
              "prepare": {
                "split": 0.2,
                "seed": 20170428
              },
              "featurize": {
                "max_features": 3000,
                "ngrams": 2
              },
              "train": {
                "seed": 20170428,
                "n_est": 100,
                "min_split": 64
              }
            }
          }
        },
        "queued": false,
        "running": false,
        "executor": null,
        "metrics": {
          "scores.json": {
            "data": {
              "avg_prec": 0.6040544652105823,
              "roc_auc": 0.9608017142900953
            }
          }
        },
        "name": "master"
      }
    },
    "5d1639dc071d8f76c9655fcbaa1a97aef87392f1": {
      "data": {
        "timestamp": "2021-10-05T21:55:17",
        "params": {
          "params.yaml": {
            "data": {
              "prepare": {
                "split": 0.2,
                "seed": 20170428
              },
              "featurize": {
                "max_features": 2000,
                "ngrams": 2
              },
              "train": {
                "seed": 20170428,
                "n_est": 100,
                "min_split": 64
              }
            }
          }
        },
        "queued": false,
        "running": false,
        "executor": null,
        "metrics": {
          "scores.json": {
            "data": {
              "avg_prec": 0.5858888885424922,
              "roc_auc": 0.944996664954421
            }
          }
        },
        "name": "exp-8dd71"
      }
    },
...

in this example 00071e is the original commit SHA (master) and 5d1639 is the exp SHA (exp-8dd71) So diffing 00071e.baseline.data.params and 00071e.5d1639.data.params (using the full SHAs as keys) should give you the same information you get from params diff (and the same for data.metrics and metrics diff)

[pmrowla]

Upon re-reading, maybe I'm misunderstanding the question. Are you asking for the workspace exp show entry to include both the data for the current workspace + HEAD?

The one thing to note here would be that if you run dvc exp show --show-json with no other args, the only returned git commit is always HEAD, so you can just diff workspace.baseline.data against <git_sha>.baseline.data

[mattseddon]

Thanks @pmrowla, I now feel stupid that I missed something so obvious.

How do both params diff & metrics diff get the previous commit's SHA (not HEAD) when an experiment is running? I thought that maybe I could just run git rev-parse HEAD but that doesn't seem to be the case.

[pmrowla]

HEAD moves during experiment execution (to point to the latest exp/checkpoint commit. During experiment execution (in the workspace), refs/exps/exec/EXEC_BASELINE will point to the original (previous commit), so you need to use that ref instead of HEAD during experiment execution.

In DVC, when we lookup HEAD in commands like exp show or params/metrics diff we check if the exp exec refs exist and use them in place of refs/HEAD when they exist:

dvc/dvc/repo/experiments/utils.py

Lines 113 to 118 in d6ecef7

	def fix_exp_head(scm: "Git", ref: Optional[str]) -> Optional[str]:
	if ref:
	name, tail = Git.split_ref_pattern(ref)
	if name == "HEAD" and scm.get_ref(EXEC_BASELINE):
	return "".join((EXEC_BASELINE, tail))
	return ref

(so during execution, the single commit returned by exp show is technically refs/exps/exec/EXEC_BASELINE and not refs/HEAD)

[dberenbaum]

Regardless of how you are implementing it, highlighting changes in metrics and params is a nice idea for exp show and a good potential alternative to #5966. @daavoo What do you think about this as a table improvement?

[daavoo]

Regardless of how you are implementing it, highlighting changes in metrics and params is a nice idea for exp show and a good potential alternative to #5966. @daavoo What do you think about this as a table improvement?

I think it makes sense. Created a new issue. Should we close this as the original question seems resolved (@mattseddon) ?

[mattseddon]

Thanks @pmrowla, @dberenbaum & @daavoo.

Happy to close this issue.

In the interim I can copy the DVC implementation. However, IMO "external products" should not be relying on the internals of DVC and/or reimplementing parts of the DVC code base. To avoid both of those things we would need this data to be exposed in some way through exp show or continue to use dvc params diff & dvc metrics diff. Using the extra commands causes more problems than it solves that's why I raised the issue originally.

Can we add the external-product label to #6783?

[pmrowla]

Showing/highlighting deltas in the exp show table can already be implemented as UI on top of the existing available data in both DVC and the vscode extension (since both the original and new params/metrics values are already exposed in the existing exp show --show-json data).

I think @mattseddon is asking for the deltas to also be included in the json in addition to the existing data, which IMO is a separate discussion from any spinoff issue for displaying the deltas in the DVC CLI table (#6783)

---

On re-implementing DVC internals, handling the git ref mechanics of HEAD vs EXEC_BASELINE should be unnecessary on the vscode extension side (since exp show already accounts for it, and returns all the data needed in place of the original 3 commands that vscode was using)

[mattseddon]

Thanks for clarifying. I clearly missed that under normal circumstances the data (dict) only contains 2 top level keys, one for workspace and another for the current commit. I thought that I was going to have to check the refs to see which one to use. Thanks 👍🏻 .