`exp show`: differentiate params, metrics, data

[dberenbaum]

After #7089, there are now 4 different sets of columns in dvc exp show each with its own designated color:

1. General experiment info
2. Metrics
3. Parameters
4. Data

It would be helpful to more clearly designate which columns fall into each category. For example:

• A cross-column "metrics" heading (see [REQUEST] table multiple multi colum headers Textualize/rich#1529).
• Prefix each column with a letter designating its column type, like m:avg_prec.

We don't even document the color or order of columns in exp show today AFAIK. Colors may be unsupported or hard to see (not sure if a colorblind-friendly palette was considered), in which case the colors become insufficient.

It could also be helpful for filtering. For example, if columns were designated like d:data.csv, it would be easy to drop all data columns with --drop 'd:.*', especially since filtering by metrics or parameters was removed in #7141.

[shcheklein]

Hey @iterative/websites and @iterative/vs-code let's pay attention to this since it already affects us (data columns to show + the future changes)

@dberenbaum one thing also to look into. dvc exp show CLI table doesn't have any "layers" like Studio and VS Code have ... without them it's hard to support more or less real nesting (and keep table sane width), also hard to disambiguate when there are multiple files

[daavoo]

There seem to be a few differe issues here:

We don't even document the color or order of columns in exp show today AFAIK.

👍 Will try to cover as part of the docs P.R. for #6434

Colors may be unsupported or hard to see (not sure if a colorblind-friendly palette was considered), in which case the colors become insufficient.

I believe this should be revisited by the design team as it affects tables cross-product and a common palette should be defined for all.

It could also be helpful for filtering. For example, if columns were designated like d:data.csv, it would be easy to drop all data columns with --drop 'd:.*', especially since filtering by metrics or parameters was removed in #7141.

This filtering logic could be implemented internally without the need to be reflected in the rendering of the table, just mentioning so that it doesn't bias the solution for better differentiating.

A cross-column "metrics" heading (see [REQUEST] table multiple multi colum headers Textualize/rich#1529).
CLI table doesn't have any "layers" like Studio and VS Code have

This seems to be an unsolved limitation of rich: Textualize/rich#164

[dberenbaum]

This filtering logic could be implemented internally without the need to be reflected in the rendering of the table, just mentioning so that it doesn't bias the solution for better differentiating.

Good point, although it means --keep/--drop no longer exactly reflect the column names. Not sure what's best honestly.

This seems to be an unsolved limitation of rich: Textualize/rich#164

Also note that it's open to sponsorship.

[pmrowla]

The color palette as it currently exists was just kept from the original "we could do this to kind of make the CLI table look like the viewer" example, from way back before experiments were released.

I believe this should be revisited by the design team as it affects tables cross-product and a common palette should be defined for all.

This makes sense for actual GUI applications like studio + vscode, but I'm not sure we should try to enforce this on the CLI side.

Honestly I think we should probably get rid of having any kind of background color usage at all in the CLI, since there's always the potential it ends up unreadable depending on the user's terminal config (see also: the recent light/dark theme discussions wrt alternating row bgcolors)

---

On differentiating data in the table, I noticed that we are starting to show a lot of different types of hashes in the table now (git commit SHAs, exp- hashes, DVC data hashes). It might be good for us to include the <name>: prefix in the DVC data columns to make it easier to differentiate them. So in data/features we could have entries like md5:abc123, md5:def456, etag:ghi789 instead of just "arbitrary short hash that looks exactly the same as a short git commit SHA"

[dberenbaum]

dvc exp show CLI table doesn't have any "layers" like Studio and VS Code have ... without them it's hard to support more or less real nesting (and keep table sane width), also hard to disambiguate when there are multiple files

This seems to be an unsolved limitation of rich: Textualize/rich#164

Yes, having nested columns would also allow us to shorten column widths for paths in nested directories, which would be really helpful even if we don't use it to differentiate params, metrics, and data. Maybe we could look into sponsoring or contributing back?

Even without true nesting, we could start splitting column headers to multiple lines to condense width (for example, on path separators).

On differentiating data in the table, I noticed that we are starting to show a lot of different types of hashes in the table now (git commit SHAs, exp- hashes, DVC data hashes). It might be good for us to include the <name>: prefix in the DVC data columns to make it easier to differentiate them. So in data/features we could have entries like md5:abc123, md5:def456, etag:ghi789 instead of just "arbitrary short hash that looks exactly the same as a short git commit SHA"

Good point. This can differentiate from both the other columns and other hashes. IMO a suffix ab1:md5 might be slightly nicer because it's harder to visually parse the differences between rows when they have the same first characters. It's one reason hashes can be easier to distinguish than file sizes.

I also think we can shorten these hashes, both because md5 will add to column width and because shorter values are easier to parse visually. Even 3 characters might be enough here, since there's still only 1/4096 chance of collision for md5 and the worst case scenario is mistakenly thinking 2 deps are the same.

─────────────────────────────────────────────────────────────────────────────────────────
n.py   src/evaluate.py   data/features   data/data.xml   model.pkl     data/prepared   sr
─────────────────────────────────────────────────────────────────────────────────────────
549    md5:fb7b520       md5:52c1fdd     md5:a304afb     md5:b7be5d0   md5:20b786b     md
549    md5:fb7b520       md5:52c1fdd     md5:a304afb     md5:b7be5d0   md5:20b786b     md
549    md5:fb7b520       md5:52c1fdd     md5:a304afb     md5:484fab5   md5:20b786b     md
549    md5:fb7b520       md5:52c1fdd     md5:a304afb     md5:b7be5d0   md5:20b786b     md
549    md5:fb7b520       md5:f237c73     md5:a304afb     md5:17b3d1e   md5:20b786b     md
549    md5:fb7b520       md5:2b5e0fd     md5:a304afb     md5:c4c0670   md5:20b786b     md
549    md5:fb7b520       md5:2b5e0fd     md5:a304afb     md5:c4c0670   md5:20b786b     md

tion.py   src/prepare.py   src/train.py   data/data.xml   model.pkl   data/prepared   dat
─────────────────────────────────────────────────────────────────────────────────────────
          515:md5          9ab:md5        a30:md5         b7b:md5     20b:md5         52c
          515:md5          9ab:md5        a30:md5         b7b:md5     20b:md5         52c
          515:md5          9ab:md5        a30:md5         484:md5     20b:md5         52c
          515:md5          9ab:md5        a30:md5         b7b:md5     20b:md5         52c
          515:md5          9ab:md5        a30:md5         17b:md5     20b:md5         f23
          515:md5          9ab:md5        a30:md5         c4c:md5     20b:md5         2b5
          515:md5          9ab:md5        a30:md5         c4c:md5     20b:md5         2b5

[dberenbaum]

Closing as not actionable ATM