`plots`: standardise across DVC/Studio/VS Code

[mattseddon]

Summary / Background

Discussions regarding migrating the underlying plots' engine from Vega to Plotly were recently re-started (see treeverse/dvc-render#7) due to several plots-related issues appearing in vscode-dvc (e.g. treeverse/vscode-dvc#4530, treeverse/vscode-dvc#4532, treeverse/vscode-dvc#3837 and there are more). Initial scoping of that issue (re-)highlighted the current difference between the three product's UI/data processing. These three images show the "same" plot across the 3 products:

An initial step in scoping the migration effort was to implement zoom/pan functionality to Vega plots within the VS Code extension (treeverse/vscode-dvc#4629). The immediate feedback provided was "But this needs to be available in Studio too!". Again (and unfortunately), that is no simple task because all three products take such different approaches. A simple template update can not simply be plugged in.

Another example of unified changes being difficult was plots per step being added in both Studio (https://github.com/iterative/studio/pull/7342) + VS Code (treeverse/vscode-dvc#4372). It should also be noted that the slider is currently unavailable via the CLI output.

In order to make unified changes across the product easier we need to standardise.

Scope

As a first step to fixing the above problems, we are going to standardise the appearance of plots provided by built-in dvc-render templates across the three product's UIs.

Forcing the UIs to converge will give insights into the code. Those insights will enable us to refactor and move towards a standardised approach for dealing with plots/their data. As we remove legacy code/processes we will be able to iterate quicker on plot changes and not get bogged down every time we want to do something as simple as change a template.

At a minimum, we should display the same legend for each plot and we should also have the same underlying data in each plot.

Assumptions

• The overall approach is up for debate.
• There is business appetite to be able to iterate quicker in the future.
• There is business appetite to have three standardised product UIs.
• Do not need to attempt to improve the HTML produced by the CLI.
• Usage of custom templates is low enough not to worry about being backwards compatible with user-defined custom templates.

Open Questions

• Can we split out parts of dvc-render into JSON + Typescript + move into DVC? 👈🏻 This should become clearer as we get further into the process.
• Can top-level bar plots have multiple data sources? How should they be treated?

Blockers / Dependencies

None yet.

General Approach

(after discussing with @dberenbaum)

• Match CLI/Studio appearance to VS Code plot.
• Add extra template anchors to the baseline templates and docs. These will include color, stroke dash and shape mappings along with a field to group data (equivalent of the current rev field used by DVC to render plots). I.e. <DVC_METRIC_COLOR> / <DVC_METRIC_STROKE_DASH> / <DVC_METRIC_SHAPE> / <DVC_METRIC_GROUP_BY>
• Fill the anchors in each of the products based on.
• Repeat colors for the CLI when > 7 have been selected.
• Delay combining code for now.
• Add <DVC_METRIC_ROW> for bar charts and only use <DVC_METRIC_SHAPE> for scatter plots.

Steps

• Propose new template format (plots: standardise across DVC/Studio/VS Code dvc-render#142)
• dvc-render changes to render new templates via CLI (plots: standardise across DVC/Studio/VS Code dvc-render#142)
• DVC changes to render new templates (plots: standardise across DVC/Studio/VS Code #9931)
• DVC changes to provide updated data to VS Code (plots: standardise across DVC/Studio/VS Code #9931)
• VS Code changes to accept new template and data (Bump min DVC version to 3.33.0 (standardise plots across products) vscode-dvc#4734)
• Studio changes to update data processing and UI (https://github.com/iterative/studio/pull/8242 + https://github.com/iterative/studio/pull/8301 + https://github.com/iterative/studio/pull/8340)
• Document changes (plots: standardise across DVC/Studio/VS Code dvc.org#4978)
• Update dvc-render in dvclive (plots: standardise across DVC/Studio/VS Code (update dependencies) dvclive#747)

Note: Took a detour to add tests to Studio and do some refactoring before attempting to make the required changes. All the PRs are linked to this issue.

[mattseddon]

cc @iterative/studio-frontend @tapadipti @daavoo.

[daavoo]

Does the approach have to be backwards compatible with user-defined custom templates?

I think the usage is too low to be worth the effort.
Also, many custom templates are most likely broken in some way if you compare the 3 UIs today.

[daavoo]

Can we split out parts of dvc-render into JSON + Typescript + move into DVC? 👈🏻 This should become clearer as we get further into the process.

I would love it if DVC takes only care of collecting data sources and definitions (what is basically today's internal Repo API) and leaves the rendering (including building and defining templates) to the UIs.

I want to also give some background context on 2 parts:

• The local HTML was created back when there were no other UIs.
• dvc-render was created so DVCLive could generate live updates while the experiment was running, without depending on DVC (there was a time when the push was to keep DVCLive usable on its own). It was meant to be used by DVC, DVCLive, and Studio backend to avoid duplicating logic.

We now have the VSCode extension and Studio UIs, both supporting live updates.
Studio backend uses dvc-render but only partially while still using custom code and introducing inconsistencies.

I see the existence of the local HTML and dvc-render as legacy corner cases, and would be happy to deprioritize any changes there / drop them.

[mattseddon]

Thanks @daavoo

I would love it if DVC takes only care of collecting data sources and definitions (what is basically today's internal Repo API) and leaves the rendering (including building and defining templates) to the UIs.

This is where we are heading.

I see the existence of the local HTML and dvc-render as legacy corner cases, and would be happy to deprioritize any changes there / drop them.

I'll see how difficult making updates there are going to be. I'm hopeful that we won't need to free or abandon rendering plots with the CLI.

[shcheklein]

I would love it if DVC takes only care of collecting data sources and definitions (what is basically today's internal Repo API) and leaves the rendering (including building and defining templates) to the UIs.

(just to start the discussion, probably a bit too early, fine to ignore for now)

Can we list the implications of this decision (e.g. each product will have to implement an engine, each product will have its own set of definitions for plots, anything else?). How critical it is for us?

Btw, for basic plots I think we don't even need templates as something users deal with - we expect them to work out of the box. So, it all comes back to customizations, I guess?

[mattseddon]

Can we list the implications of this decision (e.g. each product will have to implement an engine, each product will have its own set of definitions for plots, anything else?). How critical it is for us?

If we separate the concerns in that way we can have a shared Typescript package which can be imported by both VS Code and Studio. It will be responsible for processing/combining the data and working out the appropriate groups (see next comment) before it can be fed into the different app's components. I am also hoping to reuse that package for creating the current index.html. At this point, this is only an idea and if we do have a shared package it will bring in another set of problems.

Btw, for basic plots I think we don't even need templates as something users deal with - we expect them to work out of the box. So, it all comes back to customizations, I guess?

One reason for having default templates/making them visible is that (hopefully) users can find them and customise them to their own needs. I do however know that at this stage there is no evidence to support the idea that regular users will self-serve in this way.

[mattseddon]

For the 4 template groups this is what we are aiming at:

Linear:

Note the updated tooltip. The plot can potentially contain different shapes when the data has multiple fields in multiple files.

No change to the simple template.

Bar:

In the above example (and whenever there are different data sources i.e. filename::field combos) we will introduce the row variable and stack groups on top of each other in separate plots.

Potentially we could introduce a column instead of a row.

Please let me know if you have any strong opinions on this.

Scatter:

Confusion Matrix:

No change for now but I will be looking into migrating to Plotly Heatmap as data is aggregated prior to rendering. The main benefit would be that we will have to send much much less data over the wire (as JSON). We may also be able to unmerge revs as we can specify a custom/standardised color scale for different plots. This would potentially give us a discrepancy in zoom + pan behaviour but we can weigh up the tradeoffs when we get to them.

[mvshmakov]

If we separate the concerns in that way we can have a shared Typescript package which can be imported by both VS Code and Studio.

+1 for it, shared codebase will help the standardization + allow us to not deviate from the shared logic as we go forward + implement it once instead of several times. It has its own set of problems, but IMO is worth it for the complex plots problem.