experiments: parallel coordinates plot

[dmpetrov]

I'd love to see a parallel coordinates plot for an experiment.

$ dvc exp show --no-pager --include-metrics accuracy --include-params epochs,opt.lr --all-commits
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━┓
┃ Experiment                        ┃ accuracy ┃ epochs ┃ opt.lr ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━┩
│ workspace                         │  0.48982 │ 2      │ 0.0    │
│ 794e714 (2020-08-24 03:01:44)     │  0.48982 │ 2      │ 0.0    │
│ ├── a0f6717 (2020-08-24 03:17:31) │  0.84703 │ 2      │ 1e-05  │
│ ├── 59b2759 (2020-08-24 03:12:42) │  0.86678 │ 2      │ 1e-05  │
│ ├── 26656e0 (2020-08-24 03:16:02) │  0.67587 │ 2      │ 0.0    │
│ └── 0a81605 (2020-08-24 03:16:42) │  0.81715 │ 2      │ 1e-05  │
│ 650049c (2020-08-21 20:43:31)     │  0.48607 │ 2      │ -      │
...
$ dvc exp plots 794e714 # show all ephemeral commits of a given commit as a  parallel coordinates plot 
file:///..../plot.png

$ dvc exp plots a0f6717 59b2759 ba39f4  # given set of ephemeral commits
file:///..../plot.png

$ dvc exp plots a0f6717 59b2759 ba39f4 --order epochs,opt.lr,accuracy  # change the order
file:///..../plot.png

[pared]

Seems like it will involve creating a new template.

[pmrowla]

See https://vega.github.io/vega-lite/examples/parallel_coordinate.html for example parallel coordinates plot schema

This will also require adding support for parameters in DVC plots, my understanding is that we cannot currently plot values from params.yaml since it is not a DVC out
@pared can you confirm this?

[pared]

@pmrowla That is correct, currently we are using find_outs_by_path to gather plot data. We probably should generalize it to accept any viable file.
Related: #4446

[pared]

NOTE: accepting any viable file is already implemented in #4446

[daavoo]

Would this belong in a separated plots command or could it be an optional flag in dvc exp show?

It feels like parallel plots don't make much sense without experiments and the implementation would probably be easier if we handle the generation as an optional flag of dvc exp show like the existing --show-json and --show-csv. Something like --show-plot?

[daavoo]

So after some research here is my idea on how to implement a minimum viable version:

• Add a vega-lite template (adapted from https://vega.github.io/vega-lite/examples/parallel_coordinate.html)

Tabular datasets can and should be able to use this kind of plot (as shown in the example vega lite where raw tabular data is plotted).

For the case of experiments, the template would be used in dvc exp show to render data extracted from the internal TabularData (not clear how yet, i.e. calling directly dvc.render.write?).

The internal TabularData contains all the information required to fill the data anchor of the template (including metrics, params and revisions), and this would make the options already existing for dvc exp show to be applied to the parallel coordinates plot (i.e. --include-metrics, etc.)

This plot doesn't fit the usual schema of anchors we use for other plots . For this kind of plot, we need to have anchors for: raw data; multiple columns to display (fold field in the linked template); property for encoding color (Species used in the example).

This feels kind of blocked by #5980 although there could be workarounds with the existing functionality.

• Extending TabularData functions.

After trying different external tools for generating parallel coordinate plots, it looks that the better way would be to extend TabularData with some functions to clean the data before generating the plot. The following functions are usually applied automatically to the data in many external tools:

1. Dropping NaNs (Need to decide on defaulting to cols and/or rows). At least for rendering in vega, dropping fees like the easiest way to handle NaNs. Other options feel tricky to implement in the vega template. TabularData: Add dropna method #6812

2. Dropping duplicates (Need to decide on defaulting to cols and/or rows). Implementing this in TabularData and exposing it as a CLI arg could fix exp show: show only changed parameters #5966 (would be cols for that case).

3. Dropping categorical (non-scalar) columns. For some tools, there is support or workarounds for mixing scalar and categorical (i.e. plotly) . For vega template supporting this feels tricky so again feels like dropping would be the fastest way to minimum viable version. Currently TabularData does not store information about the original types so it should also be extended to store the type associated with each column.

CC @pared @dberenbaum