exp show: --show-md for experiment tables

[mattlbeck]

We want to be able to display tables of our experiments in markdown documents similar to how we can do it with dvc params and metrics.

Our use case is that we are conducting hyperparameter tuning jobs in CML, and would like to easily format a table of the results in PR comments.

I appreciate that just doing a dvc exp show --show-md could by default be an overwhelming table but the conversion would be extremely handy along with the filtering flags to get a summary that is much more manageable.

[dberenbaum]

Good idea, @mattlbeck!

I'm curious if you see benefits beyond ease of use for doing this over inserting the output directly as a code block like:

echo '```' >> report.md
dvc exp show >> report.md
echo '```' >> report.md

[mattlbeck]

@dberenbaum Honestly hadn't thought of placing inside of a code block. Presumably this only works with --no-pager?

Without having properly tested this, the only additional benefit of a --show-md I can think of is that it would look a bit nicer.

[dberenbaum]

Good to know. Would you be interested in either trying that workaround and letting us know how it works, or else contributing the --show-md option?

[mattlbeck]

No problem, I will either submit a PR or close this ticket depending on the outcome.

[dberenbaum]

Thank you! One more thought: the leftmost column condenses multiple items of info (compare to the csv output) that might be hard to show in the same way in markdown.

[dberenbaum]

Also, no need to close the ticket. This is at least a pattern we should explicitly support for cml and other ci needs.

[daavoo]

Raw markdown would be more likely to benefit from some features of the platform where the table is being rendered (i.e. in Jupyter you would get rows highlighted on hover).

Taking Github example below, I kind of like the --show-md format better.

Example "code block workaround" (this renders awful on VSCode markdown extension preview, btw):

┏━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Experiment ┃ Created      ┃    loss ┃ accuracy ┃ train.batch_size ┃ train.hidden_units ┃ train.dropout ┃ train.num_epochs ┃ train.lr ┃ train.conv_activation ┃
┡━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━┩
│ workspace  │ -            │ 0.26484 │   0.9038 │ 128              │ 64                 │ 0.4           │ 10               │ 0.001    │ relu                  │
│ main       │ Sep 14, 2021 │ 0.26484 │   0.9038 │ 128              │ 64                 │ 0.4           │ 10               │ 0.001    │ relu                  │
│ 5bcd44f    │ Sep 01, 2021 │ 0.25026 │   0.9095 │ 128              │ 64                 │ 0.4           │ 10               │ 0.001    │ relu                  │
│ b06a6ba    │ Aug 31, 2021 │ 0.25026 │   0.9095 │ 128              │ 64                 │ 0.4           │ 10               │ 0.001    │ relu                  │
│ d34fd8c    │ Aug 30, 2021 │ 0.30741 │   0.8929 │ 128              │ 64                 │ 0.4           │ 10               │ 0.01     │ relu                  │
│ 02b68b7    │ Aug 29, 2021 │ 0.44604 │   0.8483 │ 128              │ 64                 │ 0.4           │ 10               │ 0.01     │ relu                  │
│ 5337519    │ Aug 28, 2021 │       - │        - │ -                │ -                  │ -             │ -                │ -        │ -                     │
└────────────┴──────────────┴─────────┴──────────┴──────────────────┴────────────────────┴───────────────┴──────────────────┴──────────┴───────────────────────┘

Pure markdown:

Experiment	Created	loss	accuracy	train.batch_size	train.hidden_units	train.dropout	train.num_epochs	train.lr	train.conv_activation
workspace	-	0.26484	0.9038	128	64	0.4	10	0.001	relu
main	Sep 14, 2021	0.26484	0.9038	128	64	0.4	10	0.001	relu
5bcd44f	Sep 01, 2021	0.25026	0.9095	128	64	0.4	10	0.001	relu
b06a6ba	Aug 31, 2021	0.25026	0.9095	128	64	0.4	10	0.001	relu
d34fd8c	Aug 30, 2021	0.30741	0.8929	128	64	0.4	10	0.01	relu
02b68b7	Aug 29, 2021	0.44604	0.8483	128	64	0.4	10	0.01	relu
5337519	Aug 28, 2021	-	-	-	-	-	-	-	-