feat: Add observable10 palette #3843

mcnuttandrew · 2023-12-12T18:51:07Z

Announced here observable has a new color palette called observable10. I don't have a great sense if this type of keeping of with the color-pal-joneses would be seen as useful here, but I figured I'd add it and see what you all thought.

domoritz · 2023-12-12T18:57:52Z

Screenshots. Looks nice to me. Let's add it. Do you know whether the colors are the same for dark and light mode or whether these are two palettes? Looks like one.

domoritz · 2023-12-12T18:58:35Z

Before we merge, I'd like to get some sign off from other steering committee members.

mcnuttandrew · 2023-12-12T20:51:50Z

I went into the code to double check if its just one pal---it does appear to be, see here---and realized my image-based measurement of the colors was slightly incorrectly, colors should now be right

domoritz · 2023-12-12T22:16:36Z

Can we pull in d3/d3-scale-chromatic#51 directly? Otherwise we should check for license/attribution requirements.

arvind · 2023-12-12T22:52:59Z

Odd question but: would the Observable folks be happy if we were to include the color palette? I'm not sure if color palettes are licensed but, given the name, I wonder if they want their palette to be more readily identifiable with their brand/Plot? I would hate to step on their toes.

(The fact that they've contributed it back to D3 makes me think it's probably ok, but it'd be good to get their explicit approval.)

mcnuttandrew · 2023-12-12T23:03:34Z

these are fair points! Elsewhere Maureen also noted that "If you want to add it, let’s please reach out to observable and get both the correct colors and and permission. Lots of work goes into making a good general purpose palette, let’s respect that."

From a license sense, i suspect that this fully above board (https://github.com/observablehq/plot/blob/main/LICENSE) if an attribution is added. Although I'll admit im no expert in that area.
From a being community members/digital neighbors sense, yes I think it would be probably be good to check in.
If it would be preferable, I definitely can try to add support through d3-scale-chromatic, but i'd like to be sure that's what's best before going down that road given that's its not part of vega now. Pulling it in might suggest some restructuring to palette.js more generally, but would probably have the benefit that more palettes would come for free.

jheer · 2023-12-12T23:07:15Z

Let's just ask @mbostock: Mike, do you (and Observable) support adding your new obserable10 color palette as an option within Vega / Vega-Lite, without directly depending on d3-scale-chromatic?

mbostock

👍👍 Go for it! Thanks for asking.

Fil · 2023-12-13T14:52:03Z

Please hold on, we're making a slight adjustment to the palette.

domoritz · 2023-12-13T14:59:30Z

I would prefer to hold off and just import d3/d3-scale-chromatic#51 directly.

Fil · 2023-12-13T17:31:45Z

The scheme has been updated (reordered) in plot 0.6.13.

Per our CHANGELOG:

In 0.6.12, we inadvertently published an earlier draft; this fixed version better aligns the order of colors with tableau10.

The correct scheme (reference below) is identical to d3-scale-chromatic#51. Please, update this PR to match (or follow @domoritz and use d3-scale-chromatic).

I want to thank @mcnuttandrew and the team here for your enthusiasm and reactivity. It's by reviewing this PR that I noticed the discrepancy, and our mistake! (And fortunately it was not too late to hot-fix.)

(Note the green in position 4.)

jheer · 2023-12-13T23:04:23Z

For simplicity and to avoid bloat I would recommend that we not add a new dependency in this PR -- just add the palette within our existing setup.

If we want to add a d3-scale-chromatic dependency we should also reconsider our palette system. At the moment (for better or worse) a direct replacement will "break" some of the adjustments in place for Vega color schemes.

domoritz

Let's replace with an import so we are up to date with the latest colors.

mcnuttandrew · 2023-12-14T01:25:05Z

Alright with some offline help from @domoritz, I have gotten the scale-chromatic import working as requested. The version of d3-scale-chromatic that is current published (v3.0.0) does not include observable 10, and so i have just updated the stringified colors per Fil's updates. Once the new version is published we can just add a link so we'll always have the most up to date colors.

The way i'm including the scale-chromatic schemes is trying to be as un-distruptive as possible, which necessitates a slightly odd hack in which schemes from scale-chromatic are converted from hex[] back into the strings of hexes that generates them. A more holistic rethinking of how palettes work is certainly possible, but in this modestly scoped PR I'm merely trying to make it possible to keep up to date within the current architecture.

domoritz · 2023-12-14T02:48:18Z

Should we wait for the release then?

arvind · 2023-12-14T14:38:25Z

Sorry, just to double check my understanding, wasn't the preference that @jheer mentioned above to not use a d3-scale-chromatic because there are "adjustments in place for Vega color schemes"?

domoritz · 2023-12-14T14:41:16Z

Ah, I missed that. #3843 (comment). @mcnuttandrew did you make the necessary adjustments? If not, let's just copy.

mcnuttandrew · 2023-12-14T14:41:49Z

Should we wait for the release then?

Sure! I'll follow up once things d3-scale-chromatic versions

Sorry, just to double check my understanding, wasn't the preference that @jheer mentioned above to not use a d3-scale-chromatic because there are "adjustments in place for Vega color schemes"?

This PR is kinda doing both! The way scale-chromatic is added here does not change the current color support in any way. It totally could be more thoughtfully rethought into something fancier, but for now its just providing a link to the small set of schemes that can be supported without any changes to vega's color support

(i can also undo this add, im not very committed to it)

domoritz · 2023-12-14T15:35:45Z

I would prefer an import over copy but I don't know what the adjustments in place are.

keckelt · 2023-12-19T08:18:47Z

So, to recap:

The PR adds d3-scale-chromatic as a dependency,
vega color schemes that exist in d3-scale-chromatic are no longer hardcoded but imported
the imported d3-scale-chromatic schemes are converted to a string of hex values.
The difference is, that the strings are now dynamically generated (from d3) instead of being hardcoded.

However, the colors remain hardcoded in the documentation. I don't expect them to change, but if they do there would be a mismatch between the documentation and the colors actually used. How do you see this? Should we test whether the colors are still as expected?

domoritz · 2023-12-19T11:12:01Z

packages/vega-scale/src/palettes.js

@@ -62,19 +74,22 @@ export const continuous = {
  darkRed: '3434347036339e3c38cc4037e75d1eec8620eeab29f0ce32ffeb2c'
 };

+const toStream = (scheme) => scheme.map((x) => x.slice(1).toLowerCase()).join('');


Would it be cheaper to lowercase at the end after joining?

mcnuttandrew · 2023-12-19T21:24:14Z

Yep @keckelt that is an accurate summary!

However, the colors remain hardcoded in the documentation.

I've wondered about the documentation and if it might be better to auto-generate from the palettes, that way there isn't as much manual/error prone work to generate the colors for the docs. I'd be happy to write a script (in a different PR) for generating them if that'd be helpful. (although from first glance i'm not super sure where i'd write such a script as it appears that all of the docs are hard coded?)

I don't expect them to change, but if they do there would be a mismatch between the documentation and the colors actually used. How do you see this? Should we test whether the colors are still as expected?

I am confident that the colors (as manifest in this PR) are exactly the same. Here's the script I was running inside of the tests to figure out what's what.

import * as scaleChromatic from 'd3-scale-chromatic'
Object.keys(discrete).forEach((name) => {
  const vegaScheme = discrete[name];
  const chromaticName = `scheme${name[0].toUpperCase()}${name.slice(1)}`;
  if (!scaleChromatic[chromaticName]) {
    return;
  }
  const chromaticScheme = toStream(scaleChromatic[chromaticName]);
  if (vegaScheme === chromaticScheme) {
    console.log(name, chromaticName, 'matched!')
  }
});

Which returned

category10 schemeCategory10 matched!
accent schemeAccent matched!
dark2 schemeDark2 matched!
paired schemePaired matched!
pastel1 schemePastel1 matched!
pastel2 schemePastel2 matched!
set1 schemeSet1 matched!
set2 schemeSet2 matched!
set3 schemeSet3 matched!

Something like this could be added as a test to the vega-scale test suite, but it also seems weird to test a dep like that? Happy to add it if you want though.

keckelt · 2023-12-20T07:15:28Z

Something like this could be added as a test to the vega-scale test suite, but it also seems weird to test a dep like that? Happy to add it if you want though.

Yes, I also find it a bit weird. That was the only thing I noticed when I looked through the PR. That's why I wanted to point it out again.

Not critical for me.

domoritz · 2024-03-12T16:51:42Z

@mcnuttandrew could you wrap up this pull request since there has been a lot of work around it already?

mcnuttandrew · 2024-03-12T16:57:35Z

my understanding is that i was blocked by d3/d3-scale-chromatic#51 landing the new observable 10

domoritz · 2024-03-12T17:03:20Z

Should we make in the stream until that's released? I think the scheme doesn't change often if at all.

mcnuttandrew · 2024-03-12T18:17:33Z

I am not entirely convinced that its true that it won't change, which I suspect is why it's idling.

In any case, I'm not sure what else needs to be done to wrap it up? I think i addressed the requested changes? but maybe im missing something obvious?

Fil · 2024-03-12T18:23:24Z

It will not change, but we need to take a moment to cut a new release (of d3-scale-chromatic but also of its dependents: d3, Plot).

domoritz · 2024-03-12T18:44:41Z

What's the performance impact of running toStream for every scheme? If we put the compiled strings in the code, we don't have to run the method every time Vega is loaded. But maybe the performance impact is super tiny.

It's probably tiny given that the method is just

const toStream = (scheme) => scheme.map((x) => x.slice(1)).join('').toLowerCase();

mcnuttandrew · 2024-03-12T18:59:05Z

It appears to be trivial. it's a little hard to test because it is already so fast, but it appears add about 5 milliseconds (measured from within the test suite). (although it may be faster still with out testing)

domoritz · 2024-03-12T20:40:55Z

Okay. Then let's wait for d3/d3-scale-chromatic#51 and then merge this.

mbostock · 2024-03-13T00:23:43Z

d3-scale-chromatic 3.1.0 has now been published.

mcnuttandrew · 2024-03-13T00:43:19Z

@domoritz i think this is ready to go now!

domoritz · 2024-03-13T17:20:32Z

packages/vega-scale/src/palettes.js

 export const discrete = {
-  category10: '1f77b4ff7f0e2ca02cd627289467bd8c564be377c27f7f7fbcbd2217becf',
+  accent: toStream(schemeAccent),
+  category10: toStream(schemeCategory10),


I think it's funny that we are basically reversing the color generation from https://github.com/d3/d3-scale-chromatic/blob/2c52792197299346b7bdb94322bb4dff8f554fea/src/categorical/category10.js#L3.

Small warning here: these are currently hexadecimal colors, but the format might change in the future. I don’t think in practice we will change them, but I don’t want to commit to never changing them. And certainly we might introduce new color schemes in the future that using a different format (e.g., to use the wider p3 color space).

I think it's funny that we are basically reversing the color generation

this is an irony very much not lost on me :)

Small warning

That's a very good point, but my intuition is that's a bridge that can be crossed when it's encountered. What would be the right way to guard against this change? Including a note saying to look out for that? Include a check in toStream that throws if it encounters not hex characters?

Yeah, we can cross that bridge when we come to it.

For now, I wonder whether we can remove the toStream method and recognize if we have an array of colors in the palette and skip the colors method in schemes.js.

ok that appears to be fine

domoritz · 2024-03-13T21:23:49Z

Thank you @mcnuttandrew

@domoritz

**docs** * remove Google Analytics. (via #3894) (Thanks @domoritz!) **monorepo** * switch to ESLint flat config file. (via #3920) (Thanks @lsh!) * update node versions in ci. (via #3915) (Thanks @domoritz!) **vega-encode** * use domainMin and domainMax to set scale padding. (via #3906) (Thanks @Jami159 & @lsh!) **vega-scale** * Add observable10 palette. (via #3843) (Thanks @mcnuttandrew!) * Respect tickMinStep for binned scale. (via #3904) (Thanks @alexxu-db!) **vega-scenegraph** * Revert TooltipHideEvent to MouseOutEvent to fix events on mobile. (via #3909) (Thanks @kadamwhite!) **vega-typings** * Add observable10 palette. (via #3843) (Thanks @mcnuttandrew!) **vega-view** * turn off all handlers in View.finalize() to fix memory leak. (via #3896) (Thanks @cmerrick!)

mcnuttandrew changed the title ~~Add observable10 palette~~ feat: Add observable10 palette Dec 12, 2023

domoritz approved these changes Dec 12, 2023

View reviewed changes

domoritz requested review from arvind, kanitw and jheer December 12, 2023 18:58

mbostock approved these changes Dec 13, 2023

View reviewed changes

domoritz requested changes Dec 13, 2023

View reviewed changes

mcnuttandrew added 2 commits December 13, 2023 16:45

Add observable 10

6264d20

adjust colors

f5f6b1e

mcnuttandrew force-pushed the observable-10 branch from 29863c1 to f5f6b1e Compare December 14, 2023 01:12

mcnuttandrew requested a review from a team as a code owner December 14, 2023 01:12

mcnuttandrew added 2 commits December 13, 2023 17:13

add scale-chromatic as import

150dd0e

lint

ee753f2

domoritz reviewed Dec 19, 2023

View reviewed changes

mcnuttandrew added 2 commits December 19, 2023 13:08

adjust lowering

a9ca197

.

3f38ce2

use the new scheme

9cb3c6a

domoritz reviewed Mar 13, 2024

View reviewed changes

mcnuttandrew and others added 2 commits March 13, 2024 11:08

.

b5f52c3

use vega utils

edf64dd

domoritz enabled auto-merge (squash) March 13, 2024 21:04

domoritz merged commit d828dff into vega:main Mar 13, 2024
4 checks passed

lsh mentioned this pull request May 8, 2024

chore: v5.29.0 #3921

Merged

feat: Add observable10 palette #3843

feat: Add observable10 palette #3843

Conversation

mcnuttandrew commented Dec 12, 2023

domoritz commented Dec 12, 2023 • edited

domoritz commented Dec 12, 2023

mcnuttandrew commented Dec 12, 2023

domoritz commented Dec 12, 2023

arvind commented Dec 12, 2023

mcnuttandrew commented Dec 12, 2023 • edited

jheer commented Dec 12, 2023

mbostock left a comment

Choose a reason for hiding this comment

Fil commented Dec 13, 2023

domoritz commented Dec 13, 2023

Fil commented Dec 13, 2023

jheer commented Dec 13, 2023 • edited

domoritz left a comment

Choose a reason for hiding this comment

mcnuttandrew commented Dec 14, 2023

domoritz commented Dec 14, 2023

arvind commented Dec 14, 2023

domoritz commented Dec 14, 2023

mcnuttandrew commented Dec 14, 2023 • edited

domoritz commented Dec 14, 2023

keckelt commented Dec 19, 2023 • edited

domoritz Dec 19, 2023

Choose a reason for hiding this comment

mcnuttandrew commented Dec 19, 2023 • edited

keckelt commented Dec 20, 2023 • edited

domoritz commented Mar 12, 2024

mcnuttandrew commented Mar 12, 2024

domoritz commented Mar 12, 2024

mcnuttandrew commented Mar 12, 2024

Fil commented Mar 12, 2024

domoritz commented Mar 12, 2024

mcnuttandrew commented Mar 12, 2024 • edited

domoritz commented Mar 12, 2024

mbostock commented Mar 13, 2024

mcnuttandrew commented Mar 13, 2024

domoritz Mar 13, 2024

Choose a reason for hiding this comment

mbostock Mar 13, 2024

Choose a reason for hiding this comment

mcnuttandrew Mar 13, 2024 • edited

Choose a reason for hiding this comment

domoritz Mar 13, 2024

Choose a reason for hiding this comment

mcnuttandrew Mar 13, 2024

Choose a reason for hiding this comment

domoritz commented Mar 13, 2024

domoritz commented Dec 12, 2023 •

edited

mcnuttandrew commented Dec 12, 2023 •

edited

jheer commented Dec 13, 2023 •

edited

mcnuttandrew commented Dec 14, 2023 •

edited

keckelt commented Dec 19, 2023 •

edited

mcnuttandrew commented Dec 19, 2023 •

edited

keckelt commented Dec 20, 2023 •

edited

mcnuttandrew commented Mar 12, 2024 •

edited

mcnuttandrew Mar 13, 2024 •

edited