Plots: Better confusion matrix, and normalized version to

[sjawhar]

• ❗ I have followed the Contributing to DVC checklist.

• 📖 If this PR requires documentation updates, I have created a separate PR (or issue, at least) in dvc.org and linked it here.

Thank you for the contribution - we'll try to review it as soon as possible. 🙏

Description of Changes

Relates to iterative/dvc-render#20

I really wasn't happy with the default confusion matrix provided in DVC, so I dove into Vega-Lite and figured out how to add a few features:

• Text labels on each cell showing record counts
• Added new confusion_normalized template to show counts normalized by actual label
  • I thought about adding a new anchor to control the normalization direction (groupby actual vs. predicted), but this can also easily be achieved by simply switching x and y 😄

Screenshots

New confusion template

Results of dvc plots diff with new confusion_normalized template

Feedback Wanted

Here are some stylistic decisions I made that I'm open to reverting in interest of genericness:

• Made the plot much bigger (can't reasonably overlay text on the current tiny size)
• Hid the title next to the color bar
• In the normalized plot, the scale is clamped from 0 to 1
• In the unnormalized plot, the scale minimum is clamped to 0 and the max is set to nice: true

Also, cells that are zero are empty. This is because of how the grouping works, and I couldn't find a way to "fill in" the missing values. If someone more expert in Vega syntax can help me figure that out, that would be awesome. Fixed!

[efiop]

@elleobrien @dmpetrov Would appreciate your thoughts on this 🙂

[elleobrien]

I LOVE this. It's really needed- I had a number of things I disliked about the default template that I was just modifying in my local files, and I suspect these changes will be an upgrade for a lot of data scientists.

All good from me- I really don't see any downsides. Switching the scale to normalized 0-1 also has a nice side effect of removing the awkwardly long default legend label "Count of Records", which looks bad in CML reports.

100% approve from me. Thanks @sjawhar!

[elleobrien]

Also empty cells is fine because they default to a color that isn't on the color scale- only possible improvement might be to default it to a gray that can be universally understood as NaN :)

Definitely not a blocker.

[pared]

This is a greate change @sjawhar!

Also, cells that are zero are empty.

That is not a blocker. Though I do believe this is kind of bug - scale shows that 0 is light yellow, so I guess it should be marked that way. I think we should ask Vega developers what they think about it. Since their default question channel is stack overflow we could post there. @sjawhar since you are the first to raise this issue, do you want to do that? If not, we will take care of that.

[sjawhar]

@pared I agree that it's a bug. I've posted the question on Stack Overflow

[sjawhar]

@pared @elleobrien Fixed the missing values. It should work in any case where a particular class isn't entirely missing from a column (i.e. class A was never predicted)

[pared]

@sjawhar sorry for taking the time with merging. We can merge it right away, will you solve conflicts, or should I do that?

[sjawhar]

@sjawhar sorry for taking the time with merging. We can merge it right away, will you solve conflicts, or should I do that?

No worries, I'll take care of it today.

[sjawhar]

Rebased on the latest master branch

[pared]

This is just amazing, thank you @sjawhar! 🚀 🔥

[jorgeorpinel]

Hello! Should we update docs examples per this improvement? Specifically in https://dvc.org/doc/command-reference/plots#example-confusion-matrix and https://dvc.org/doc/command-reference/plots/diff#example-confusion-matrix.

[pared]

@jorgeorpinel great point! Created PR.

[jorgeorpinel]

Ty @pared