Plots with multiple curves displayed wrong in cml

[shortcipher3]

I am getting beautiful pr curves with DVC, where I have a curve for each class on my plot.

I took what I had and created a cml pipeline and the result looked completely different/worse.

From dvc I run:

dvc plots diff

For the cml version I run:

dvc plots diff --target pr.csv --show-vega origin/main

For cml I used the docker image found on dockerhub, updated five days prior to my usage dvcorg/cml

@pared mentioned it might be due to fact that "file" results override flexible plots, as its parsed later, need to prepare a reproduction script.

[DavidGOrtega]

👋 @shortcipher3 Could you please give me insights of what CI are you using?

[DavidGOrtega]

Would be awesome also if you can provide us the metrics file

[dacbd]

I suspect I know what the issue is and have a patch. @shortcipher3 can you try using the setup-cml are or running npm install -g vega-lite@v5 in your container?

[shortcipher3]

For CI I'm using Gitlab, here is the relevant contents of my .gitlab-ci.yml

# .gitlab-ci.yml
stages:
  - cml_run

cml:
  stage: cml_run
  image: dvcorg/cml:latest
  script:
    # Setup environment
    - ln -sv $(which pip3) /usr/bin/pip3.8
    - pip install -r requirements_fiftyone.txt
    - pip install -r requirements.txt
    - npm install -g vega-lite@v5

    # Additional setup
    - git fetch --prune

    # Write any header information
    - echo "# CML Report" >> report.md

    - dvc fetch reports/model_pr.csv
    - dvc checkout reports/model_pr.csv
    - dvc plots diff 
      --target reports/model_pr.csv --show-vega origin/main > vega.json
    - vl2png vega.json model_pr.png
    - cml publish
      --driver=gitlab
      --rm-watermark
      --md model_pr.png >> report.md

    # Send the report as a comment  
    - cml send-comment report.md

My dvc.yaml looks like this:

plots:
  reports/model_pr.csv:
    template: linear
    x: recall
    y: [precision_class1, precision_class2, precision_class3, precision_class4,
      precision_class5]
    y_label: precision
    title: precision-recall @ iou 0.5

@dacbd Adding npm install -g vega-lite@v5 into my container as seen above didn't help.

@DavidGOrtega I've attached the metrics file.

model_pr.csv

[DavidGOrtega]

@dacbd Adding npm install -g vega-lite@v5 into my container as seen above didn't help.

[dacbd]

@shortcipher3 I can't exactly reproduce since we only have one version of the metrics to work with, is it possible you can provide another model_pr.csv or possibly simpler the vega.json which dvc renders properly but cml is not?

[shortcipher3]

So I just modified a couple of the numbers in the csv to get a "unique" one.

For more of a minimum reproducible experiment I ran something like:

docker pull dvcorg/cml
docker run --rm -ti -v ~:/data dvcorg/cml /bin/bash
git init .
dvc init
dvc add model_pr.csv
dvc repro test
git add dvc.yaml model_pr.csv.dvc .gitignore dvc.lock
git commit -m "test"
git checkout -b branch
# modify the file
dvc add model_pr.csv
git add model_pr.csv
git commit -m "test2"
dvc plots diff --target model_pr.csv --show-vega master > vega.json
vl2png vega.json model_pr.png

I renamed vega.json to vega.txt to upload it.
vega.txt

I renamed dvc.yaml to dvc.txt to upload it.
dvc.txt

[dacbd]

@shortcipher3 thank you for the detailed steps, I was able to determine part of the issue.

The dvc plot being rendered via dvc plots diff --targets model_pr.csv main is in the regular vega format and the plot rendered via dvc plots diff --target model_pr.csv --show-vega main > vega.json is in the vega-lite format which differs slightly

there is probably a good way to programmatically extract the vega data embed on the index.html (in which case you can use vg2png instead of vl2png), but the quickest path to something workable would be to go:

dvc plots diff --targets model_pr.csv main
cml publish dvc_plots/index.html

cml publish with output a link like: https://asset.cml.dev/36855bf8a25c5882ce4c52ec85fc5d5f9f8f272f?cml=html
which you can use in a report.md etc.

CC @daavoo @0x2b3bfa0 as feel they can probably recommend another approach?

[daavoo]

It looks like an issue with the combination of flexible plots and show_vega arg.

I believe this issue can be transferred to DVC.
There is a function (adjust_vega_renderers) being applied to the vega plot embedded in the HTML that it is not being applied to the json returned by show_vega:

https://github.com/iterative/dvc/blob/496599518a2f79a79b63888a1d9eaa30d8712021/dvc/commands/plots.py#L158-L177

[shortcipher3]

Probably a silly question, but my understanding is html tags can be used freely in markdown, so I thought, why not upload my html file.

dvc plots diff main
cml send-comment dvc_plots/index.html

This is the error I got:

{"level":"error","message":"Bad Request","stack":"Error: Bad Request\n    at Gitlab.request (/usr/lib/node_modules/@dvcorg/cml/src/drivers/gitlab.js:480:38)\n    at processTicksAndRejections (node:internal/process/task_queues:96:5)\n    at async Gitlab.commentCreate (/usr/lib/node_modules/@dvcorg/cml/src/drivers/gitlab.js:76:5)\n    at async CML.commentCreate (/usr/lib/node_modules/@dvcorg/cml/src/cml.js:307:12)\n    at async Object.exports.handler (/usr/lib/node_modules/@dvcorg/cml/bin/cml/send-comment.js:11:15)"}

Conceptually is there a reason using the html shouldn't work?

I assume the publish is uploading the data to gitlab, which I can then share a link to, but I would rather have it embedded in the comment if possible. Is there a way to do that?