log_plot(): add custom line plot

[gcaria]

During my evaluation stage, I plot a simple line plot where the x values are predefined, and the y values are calculated from truth and predicted values, and are informative of the model performance.

I'd love to see how the line plot changes from experiment to experiment, which means that I'd like to use log_plot for doing this.

Would it be possible to implement such a feature? It seems to me that log_plot offers advanced features (ROC etc) but not a simple one like this.

[dberenbaum]

During my evaluation stage, I plot a simple line plot where the x values are predefined, and the y values are calculated from truth and predicted values, and are informative of the model performance.

Could you clarify more what kind of plot you would like or give an example?

[gcaria]

Although I have a specific line plot in mind, I've tried to be as generic as possible because this is not really relevant for what I'm proposing.

In simple terms, the user would just need to specify two arrays (in a json file I guess), one for the x and one for the y coordinates of the points in a line plot (in my specific case the x values would always be the same, but they could change too, I don't expect this to be requirement).

I haven't used the available options for log_plot() because they don't apply to my case, but it seems they all have to perform some kind of computation on the json data to get the x and y values that actually go into the plot. What I'm proposing is a simpler logging option, where the user provides directly x and y (which she computed in whatever way she wants).

[daavoo]

We would need to change the current API of log_plot a little bit.

We currently use name (first arg) to select the plot template to use (https://dvc.org/doc/dvclive/api-reference/live/log_plot#supported-plots):

y_true = [0, 0, 1, 1]
y_score = [0.1, 0.4, 0.35, 0.8]
live.log_plot("calibration", y_true, y_score)

calibration above indicates both the name of the output plot and the template to use.

If we add support to this, I assume that it would make sense to support arbitrary names.
So, we would need to do 2 things:

• Introduce support for linear template
• Decouple name and introduce a new template arg (💣 breaking change)

x = [0, 1, 2]
y = [0.1, 0.2, 0.3]
live.log_plot(x, y, name="foo", tempalte="linear")

[gcaria]

My two cents on the API, admitting that I haven't seen the DVC code yet, but just trying to minimize the changes, more importantly the breaking ones:

how about just adding a new name option, which would be linear and then simply adding an optional title argument, which when provided sets the title of the plot/figure (and that would apply to all existing templates, maybe as a last step to make life easy and uniform) ?

If title is not provided then for name='linear' the title would just be e.g. Linear or Line plot

[dberenbaum]

Related: #322 (comment).

Based on that discussion, it probably makes more sense to introduce a separate method here like log_custom_plot since the current one is so sklearn-focused. The arguments can mostly follow the available configuration fields for dvc plots.

It should handle tabular (dataframe/array/tensor) or hierarchical (dict) input data (although output format could all be JSON if it's easier). Or saving the data could be separate from the plotting like in wandb. This seems a little less straightforward to me but would better support flexible plots where you want to make multiple plots from the same data source or combined data sources.

This might be a stretch for 1.0, but it would be nice to have since it sort of completes the dvc integration of being able to log any kind of dvc output from dvclive.

Like the existing log_plot method, we can start with support for no-step scenarios only, but there's some related discussion about how to support multi-step scenarios in #82.

[daavoo]

is so sklearn-focused

An additional argument towards a separate command is that, in the current sklearn plots, the inputs don't match what gets saved in the plot: (y_true, y_pred) gets transformed to some (x, y) depending on each plot.