Further plots development

[pared]

I am creating new discussion, because existing ones regarding this subject (#5808 and #5681) are focusing on singular problems we have been discussing during today's meeting.

Current state of plots

Plots, from the DVC perspective are files containing list of data points. It can be JSON, YAML, CSV or TSV file.
Example data:

epoch,accuracy,loss,val_accuracy,val_loss
0,0.9418667,0.19958884770199656,0.9679,0.10217399864746257
1,0.9763333,0.07896138601688048,0.9768,0.07310650711813942
2,0.98375,0.05241111190887168,0.9788,0.06665669009438716
3,0.98801666,0.03681169906261687,0.9781,0.06697812260198989
4,0.99111664,0.027362171787042946,0.978,0.07385754839298315
5,0.9932333,0.02069501801203781,0.9771,0.08009233058886166
6,0.9945,0.017702101902437668,0.9803,0.07830339228538505
7,0.9954,0.01396906608727198,0.9802,0.07247738889862157

The actual visual aid generation takes place upon calling dvc plots show/diff command. What DVC does:

1. Gathers data for particular plot file from one or more revision
2. Inputs the data into Vega template
3. Applies any additional config from dvc.yaml file to the plot.
4. [Optionally, true by default] wraps Vega with HTML necessary to make visualization happen locally.
   We designed plots in a format similar to our metrics command. Intended use assumes that user creates
   "raw data" files and DVC is able to provide comparison between different revisions.
   Our discussion seems to reveal that this approach (or its current implementation) does not cover all use cases that we might need to support.
   I will list the use cases that seem to be omitted and have showed up during our discussion.

Problems

1. Single data source - the data is generated, for example by grid search algorithm. The different data series that we would like to compare are in one file, but we do not have a way of telling dvc plots that. Example
2. User does not wish to use our proposed format. The plot is something unknown to us, but understandable for user - for example JPG file.
3. User provides plots as data files but does not want to use Vega. That is possible, because creating custom template is not a straightforward task and takes time. User might want his/hers own "visualization stack" to be used to visualise data.

Solutions

Potential solutions we have been talking about:

1. Provide image support - extended discussion is happening under Support for images #5681 - general idea is that we will allow users to mark image files as plots and in case of dvc plots diff given images would be just displayed one next to another. This proposition aims to solve problem 2
2. Provide API method or dvc plots command allowing user to get plots data only - since we already are gathering this data and pasting it into vega template why not let users do single command/api call and make their own plot? This proposition aims to solve 3

Some notes:

As @pmrowla noted in (#5681 (comment)): Should we focus on particular data types from DVC point of view? There are many types of image files. Do we want to support them all? And from the other point of view - what if user marks some non-image, non-data file as plot? (eg. Vega JSON)? What if user wants some other unknown format? We can always verify the plot structure/extension/whatever but the problem of format will never be solved - because we will never be able to support all use cases.

Related: #3826

Did I miss something?

[pared]

It seems that the main problem is that we dictate the way how users have to handle the plots - plots in dvc are data files that can be visualized only with Vega template. What we should be doing is provide the user with easy way to access the files needed for visualisation , no matter what kind of file it might be. We could allow user to mark any kind of data as plot and provide a way to obtain this asset via some kind of a call (think aggregated api.get_url/read) and let user handle the result. The problem with this is that user will have to handle the results by her/himself.
I think that if we will go with extending support for different file types, it might be hard to keep up with feature requests, we are already talking about diffing images on one hand and generating plots from single, structured file on the other. Abstracting the logic that would make both cases valid for dvc plots show/diff seems hard. Maybe the current state of plots should be "a suggested" way of handling visualization? DVC would provide raw data, and we could have (built in?) support for inserting it into the Vega template, but not forcing the user to do that?

[jorgeorpinel]

This is what I'm getting from @pared:

Problem (3): User does not wish to use our proposed format
Solution (2): Provide API method or dvc plots command allowing user to get plots data only
allow user to mark any kind of data as plot and provide a way to obtain this... (think aggregated api.get_url/read) and let user handle the result
For example, user could ask "give me file_x and file_y for revisions rev1, rev2, rev3" and API could return structured data

And your action-point proposal @dberenbaum:

document how iterating over api.get_url and api.open/api.read could be used to do this today

👍 It could be an example in the API index (https://dvc.org/doc/api-reference) and link from https://dvc.org/doc/command-reference/plots around where Vega is explained. Alternatively it could be a blog as it's an applied solution and may become outdated if/when this is properly implemented.

[dberenbaum]

We cover a related scenario in https://dvc.org/doc/command-reference/get#example-compare-different-versions-of-data-or-model, but it's pretty hidden at the bottom of the get command reference. We should probably highlight both the CLI and API methods together in the user guide. What do you think @jorgeorpinel?

[jorgeorpinel]

I see how that's similar to "give me file_x and file_y for revisions rev1, rev2, rev3" but looks like a completely different purpose (comparing data files vs. aggregating plots data).

highlight both the CLI and API methods together in the user guide

I'm not sure we want such a specific solution in the UG. Maybe as a how-to? If you don't mind opening an issue in the docs repo so we can finalize the plan, that would be great @dberenbaum. May be part of a larger debt we are still carrying: extract metrics/plots details from the cmd ref into the UG (there may be an issue already).

[pared]

@jorgeorpinel I guess the main point here is that this UG would be about using dvc.api to aggregate data in order to compare it in custom way, later down the road.

[jorgeorpinel]

@pared we could include it in iterative/dvc.org#2572.

But will dvc.api access plots data? Even if not cached by DVC? (assuming it's in Git) What about completely untracked ones that need to be generated with repro first, will the API use untracked plots outputs from the workspace?

I mention these scenarios bc/ I think they may be common with metrics/plots.

[dberenbaum]

Regarding support for images or other file types, I'm not sure what this would enable within DVC itself. Of course, outputs can already be images or any other file type. It seems that the only additional feature being considered is to compare images from different commits side by side. How would this be displayed in DVC? Is it worth the added UI complexity of how to define these outputs?

I'm interested in the Studio perspective on this (cc @shcheklein @tapadipti), since it seems like viewing images from different commits side by side could more naturally be shown there. Could this be handled without additional DVC support? For example, being able to select outputs with certain file extensions to compare?

[Suor]

This way any "vega template" might be inlined into such report template as well as any layouts things and also the list of things dvc need to pass to it, e.g. particular metrics, param values, plot data, file paths.

If people generate any images or whatever files they want to include they simply refer them from template and dvc will put them into place.

[Suor]

As an example to make image side by side comparison one can create a simple template like:

{% for ref in refs %}<img src="{{ref.path}}/images/demo.png">{% endfor %}

Where images/demo.png is a path to an output image presumably. Since we refer to it here we don't need to mark the output anyhow in dvc.yaml and we don't need to register report template in any way - it would be any file in some templating language, I used Jinja2 here, but that is arbitrary:

We can provide some higher level blocks for people so that they won't really need to know html much to do most things:

{% images type="side-by-side" path="images/demo.png %}

Still anything, which can be displayed in html can be displayed in a report.

[pared]

@Suor that makes more sense than trying to support different data types from inside the dvc itself.
@mnrozhkov How that would look like for your use cases?

1. GINI example - were those plots later used for some reporting? Would it make sense to create plot as matplolib image/ vega json (via altair) and include resulting image/vega json into some report? Does the process:

• create plot in code
• write plot into file
• create "report.template"
• use dvc plots to render the "report.html"
  look viable in this use case? Wouldn't it be too complicated to jump between code and template?

2. What about use case where have multiple intermediate images created during training and we want to show them all? We would probably need some glob support or including whole directory (containing the plots) into the report right?

[dberenbaum]

@Suor This seems appealing in terms of decoupling pipelines from presentation parameters. One concern is how this works if someone makes changes in dvc.yaml like changing an output name. Now do they have to repeat these changes in their presentation parameters/template?

[Suor]

If they refer a file that was moved then yes need to change it in a template too. We can try to decouple that, it will complicate things conceptually I guess.

I see two approaches and none of them perfect:

1. Make templates accept params like filename. Kinda like we do with plot templates now, i.e. and plot template may be used with any plot data file. This will be complicated if report consists of several things, i.e. some demo images side by side + some params and metrics + some trend plot - this is lots of params to template.

2. Make templates even smarter, query out names and things, may be combined with custom flags for outputs. E.g.:

{% set demo_image = find_out(glob='*/demo*.*') %}  {# find by name #}
{% set demo_image = find_out(demo_image=True) %}   {# find by custom flag #}
{% images type="side-by-side" path=demo_image.path %}

{# Display all params for preprocess and train stages, skipping seeds,
   and metrics under evaluate key plus training time in a single table #}
{% table params=find_params(stages=["preprocess", "train"], skip=r"\.seed$")   
         metrics=["evaluate.*", "train.time"] %}

[pared]

Note regarding idea of providing a way to get the plots data - If we decide to only provide the data access method, we are basically removing the ability to do a "quick" visualization - user would need own code to visualize the data. One will not want to write the code if they need, for example, just a check whether metrics is increasing and loss is going down over time.

[dberenbaum]

Yes, the data access method can be in addition to current plots, not replacing it.

[karajan1001]

In my opinion, the plotting is very custom work and involves some additional analysis. I agreed that DVC should not go too deep on providing different plotting methods, but providing some common data format. For example, I myself used to use pandas to analyze data and matplotlib to plot them.

[dberenbaum]

Based on additional comments about combining plots in iterative/studio-support#23 and #6316, does it make sense to add a target field to plots?

Here are a few examples:

Combine files

plots:
- train_loss.csv:
	x: epoch
	y: loss
	target: train_vs_val
- val_loss.csv:
	x: epoch
	y: loss
	target: train_vs_val

• dvc plots show plots:
  • train_vs_val plot with lines labeled train_loss.csv and val_loss.csv
• dvc plots diff HEAD~1 plots:
  • train_vs_val plots in facet grid by rev

---

Combine multiple columns from the same file

plots:
- val_f1.csv:
	x: epoch
	y: f1_class_0
	y_label: f1
	target: f1
- val_f1.csv:
	x: epoch
	y: f1_class_1
	target: f1

• dvc plots show plots:
  • f1 plot with lines labeled f1_class_0 and f1_class_1
• dvc plots diff HEAD~1 plots:
  • f1 plots in facet grid by rev

---

Separately plot different columns from the same file

plots:
- scores.csv:
	x: epoch
	y: acc
	target: acc
- scores.csv:
	x: epoch
	y: auc
	target: auc

• dvc plots show plots:
  • acc plot with line labeled by rev
  • auc plot with line labeled by rev
• dvc plots diff HEAD~1 plots:
  • acc plot with lines labeled by rev
  • auc plot with lines labeled by rev

---

Thoughts on how this might work:

• Need dynamic color field instead of hard coding rev
• May need to add optional facet field for default/line plots
• Collect plot configs for the target
  • Try first to label by unique path, then by y field
  • Throw errors if mismatches or missing info

[adamliter]

Are there still plans to work towards something like this? I'd really like to be able to combine metrics into a single plot along the lines of:

plots:
- train_vs_val:
    x: epoch
    y: loss
    data: [train_loss.csv, val_loss.csv]

This would be really helpful for plotting metrics for different classes in a multi-label, multi-class classification problem that I'm working on in a single plot. It seems like it might also be nice to be able to pass along something like a facet or color to Vega via something like <DVC_METRIC_FACET> or <DVC_METRIC_COLOR>, rather than having to hard code the color or facet variable name in the template like is done here.

[daavoo]

Hi @adamliter , the top-level plots should be available if you upgrade DVC to >= 2.12.1.

Docs here:

https://dvc.org/doc/command-reference/plots#top-level-plots

The syntax for your example would be:

plots:
  train_vs_val:
    y:
      train_loss.csv: loss
      val_loss.csv: loss
    x: epoch

Regarding the FACET/COLOR in Vega templates, there is an open issue here iterative/dvc-render#9 . Don't hesitate on commenting your idea / needs .

[adamliter]

Thanks @daavoo! Out of curiosity, is it possible to do this when defining a plot underneath a given stage, or only when define a plot at the top level?

[daavoo]

is it possible to do this when defining a plot underneath a given stage, or only when define a plot at the top level?

It is not possible. Only top-level plots support this syntax

[dberenbaum]

In case it's unclear, please keep in mind that you can still plot a stage output this way, you just need to define the plots in the top-level section like:

plots:
  train_vs_val:
    y:
      train_loss.csv: loss
      val_loss.csv: loss
    x: epoch
stages:
  eval:
    cmd: python evaulate.py
    deps:
      - evaluate.py
      - model.pkl
      - train_data.csv
      - val_data.csv
    outs:
      - train_loss.csv
      - val_loss.csv

[pared]

After discussing with @dberenbaum and @daavoo I am writing down some of the issues that will be impacted by this, and should be considered when implementing
#4455
iterative/dvclive#150
iterative/dvclive#82
iterative/dvclive#161

[daavoo]

Would it make sense to also consider the addition of new types of plots (https://github.com/iterative/dvc/issues/3801), given that some of those new plots might require parsing data that does not fulfill the currently expectedx/y axis schema?

Or do we consider the support of images a good enough fix for those additional types of plots?

[dberenbaum]

We could definitely support other templates. I haven't heard a lot of feedback asking for these, although it's hard to say if that's because people don't need them, they have built their own custom templates, or there aren't enough people using plots. It would be great to have some user-contributed templates, but I haven't seen that.

Most of the work here is likely coming up with the spec. Once that is complete, new templates would probably be good examples where we could use the "good first issue" label 😄 .