show: revisiting table format?

[skshetry]

Revisit table format for show.

The problem that I see with show is that they:

1. flow horizontally, which makes it harder to accommodate all the metrics/params they have, making it look like they are only optimized for toy projects.

2. Assumption that params in different metrics/params are the same and could be compared.
   eg:

    $ dvc metrics show
    Path          avg_prec    new    roc_auc
    scores.json   0.90405     -      0.80802
    metrics.json  0.60405     0.5    0.9608

   I can see the reasoning for different model builds, with similar params/metrics structure, but feels wrong to assume that by default? This makes me even more convinced that it should be vertical, rather than horizontal flow.

3. Hard to grep, as the parameter name or the metric name, are in the header, rather than in their own row like in metrics diff.

Regarding exp show, I find it overwhelming to see lots of metrics and params, which have also been shared by the users in #5966 and #6271.
Also, it's hard to grep and also flows horizontally (that too in a pager). And, it also includes metadata that might have been best suited for exp list (where we could add executors info in the future too). Anyway, I am not sure what's best here, may need to get inspiration from other tools like keepsake.

[dberenbaum]

1. flow horizontally, which makes it harder to accommodate all the metrics/params they have, making it look like they are only optimized for toy projects.

In my real projects, I have usually had a few metrics but dozens/hundreds of experiments I wanted to compare. For a single revision, vertical makes sense, but less so for comparing revisions.

2\. Assumption that params in different metrics/params are the same and could be compared.

Does vertical flow make this better? Seems the problem is that metrics show outputs a table instead of a list of metrics for each revision. exp show doesn't have this problem because it shows a list of file-metric combinations per experiment.

3\. Hard to grep, as the parameter name or the metric name, are in the header, rather than in their own row like in `metrics diff`.

For a single revision, I agree, but it's less clear to me which is better when comparing revisions.

Regarding exp show, I find it overwhelming to see lots of metrics and params, which have also been shared by the users in #5966 and #6271.

I think showing all parameters and metrics is sometimes useful, but I agree that it would be nice to make it easier to show more concise output.

Also, it's hard to grep and also flows horizontally

We could consider whether we can simplify the table to make it easier to grep (like removing borders), although some of the visual cues in the leftmost column might be hard to do in a grep-friendly way.

And, it also includes metadata that might have been best suited for exp list (where we could add executors info in the future too).

Good point. exp list could be useful for info like executors. There's also #5615, which suggests a queue command that could be useful to show queue-related info instead of in exp show. On the other hand, it is not obvious that separating info across multiple commands is always good, as users may find having it all in one table easier.

[daavoo]

Linking here to continue the discussion (from #6867 (comment)):

Looking at the current complexity of dvc exp show and our plans to add pipelines/data changes into it, maybe we are optimizing the wrong commands instead of improving exp diff and exp list?

[dberenbaum]

@skshetry Do you have example of how to make exp diff more useful? I could see including more info like data files (see #6434) and granular info about them in exp diff.

Personally, my workflow would usually be to run a bunch of experiments, compare them at a glance with something like exp show, and possibly dive deeper if needed. exp diff could be really useful in this case, but I need to already have a specific set of experiments/revisions in mind to compare, so I would need exp show to give a quick summary of all/many experiments first.

@daavoo What do you think?

[daavoo]

@skshetry Do you have example of how to make exp diff more useful? I could see including more info like data files (see #6434) and granular info about them in exp diff.

Personally, my workflow would usually be to run a bunch of experiments, compare them at a glance with something like exp show, and possibly dive deeper if needed. exp diff could be really useful in this case, but I need to already have a specific set of experiments/revisions in mind to compare, so I would need exp show to give a quick summary of all/many experiments first.

@daavoo What do you think?

I can relate to that workflow but I don't know how to make exp diff more useful. Having some additional info (i.e. data files) present in exp diff but not in exp show feels a little strange to me.

Maybe I'm biased from being used to pandas DataFrames for this kind of analysis but I only see exp diff as a reshaped view of exp show. Maybe we should actually implement exp diff that way by making it just an alias of pre-defined filtering and reshaping over exp show.

[dberenbaum]

I can think of a couple ways a separate diff command might be useful:

1. More detailed info that doesn't fit in exp show as already mentioned. What about plots, for example? Do you anticipate them eventually being included in exp show? Could they be included in exp diff?
2. Direct comparisons between two revisions that don't make sense for many revisions. Should exp diff be limited to two revisions or be able to show more? For example, exp diff already shows the amount by which metrics/params have changed. It might also show the Git diff or at least which files have changed between the two revisions.

However, I have reservations about both of these:

1. Does this make sense in exp diff? It seems more like a detailed view of an individual experiment. I would compare it to clicking on an individual experiment in an experiment tracking dashboard to see more details about it. If anything, it seems like this would make more sense as exp show and the table would make sense as exp diff.
2. How useful is it to compare two experiments like this? I think there are possibilities, but I'm not sure there are enough ideas right now to make it a priority.