`exp show`: Distinguish between different files

[dberenbaum]

@daavoo @dberenbaum

Quick ask - in Studio we can distinguish DVC-tracked and intermediate artifacts (to show only them by default), can we add those signals here too please?

Originally posted by @shcheklein in treeverse/vscode-dvc#1183 (comment)

[dberenbaum]

@shcheklein @mattseddon I'm not clear on how Studio decides what files to show. From what I can tell, it's anything with a .dvc file plus outputs of any pipeline stages.

@daavoo AFAIK the experiments table instead shows dependencies of any pipeline stages.

Do we need to reconcile these and choose one or the other? Do we need options in any of dvc, studio, or vs code to decide between them?

[daavoo]

I thought the request was to differentiate between dependencies tracked by git vs. dependencies tracked by DVC (the former are not shown in Studio table)

[dberenbaum]

@daavoo That's what I want to clarify. I see two different requests in treeverse/vscode-dvc#1183 (comment): 1) dvc-tracked files, and 2) intermediate artifacts (stage outputs).

What I see in Studio doesn't look like dependencies at all but .dvc-tracked files and stage outputs.

For example, using https://github.com/iterative/example-get-started, in Studio I see:

In dvc exp show, I see:

$ dvc exp show | head -3
 ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
  Experiment                Created        State    avg_prec   roc_auc      brier   step   prepare.split   prepare.seed   featurize.max_features   featurize.ngrams   train.seed   train.n_est   train.min_split   data/data.xml   data/features   data/prepared   model.pkl   src/evaluate.py   src/featurization.py   src/prepare.py   src/train.py
 ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

[daavoo]

Differences

• src/*

Missing in Studio - Present in DVC

These are files explicitly marked as DVC dependencies in dvc.yaml but are not shown in Studio (because they are tracked by git?).

• prc.json / roc.json / scores.json

Present in Studio - Missing in DVC

These are metrics and plots outputs (tracked by git in this case), I don't fully understand why they are treated similar to other outs files.

Exp show behavior

dvc exp show --json returns deps and outs separately but I decided to only show deps in the initial implementation because that's what I considered valuable information when comparing experiments (also to prevent noise). And output can only affect the results if it's connected to a stage as a dependency and at that point we already show them.

IMO, this makes sense semantically because users explicitly understand and define the concepts of deps and outs in dvc.yaml. The docs explicitly mention dependencies as part of the table https://dvc.org/doc/command-reference/exp/show#description

Studio behavior

I'm actually confused by Studio logic (hidden from the user):

• There is no concept or way to differentiate deps or outs
• Doesn't show git-tracked dependencies but does show git-tracked outputs
• Consider metrics and plots files as outputs, even though they have a different meaning and the actual contents are accessible as metrics / plots

---

Request

1) DVC-tracked files

The only dvc-tracked-file that is currently present in Studio but would be missing in exp show would be one that it's not connected to the pipeline at all.

• Including metrics and plots files as outputs don't make sense to me, but I can include them in the outs of --json output.

• If it's not connected to the pipeline, I'm not sure why it would make sense to be present in the exp show output, but I can include them in the outs of --json output.

• If VSCode wants to filter out git-tracked dependencies, I can include this info in the deps of --json output.

However, I personally think that the current exp show is more useful for comparing experiments than the current Studio Files.

---

2) intermediate artifacts (stage outputs)

Stage outputs are already present in --json outs section . I don't see what additional signal would be needed.

[daavoo]

@dberenbaum @shcheklein @mattseddon Can I get confirmation that you want the 3 checkboxes to be implemented and that you don't need additional info for 2) intermediate artifacts (stage outputs)?

[mattseddon]

@daavoo we'll discuss tomorrow at the cross-team meeting and come back to you, thanks.