Add Glasbey colors?

[jbednar]

Colorcet currently includes only continuous color spaces, but it was created while working on Datashader, which uses both continuous and categorical sets of colors. For rendering categorical data, Datashader requires having a distinct color per category, and it includes some ad-hoc collections of colors to try to get enough distinct colors to deal with complex datasets.

A principled approach for designing arbitrarily large sets of mutually distinct colors was presented in:

Glasbey, Chris; van der Heijden, Gerie & Toh, Vivian F. K. et al. (2007), "Colour displays for categorical images", Color Research & Application 32.4: 304-309.

"Glasbey" colors are available in ImageJ and for R, and there is a Python implementation of the method. Generating the colors with the Python code is time consuming, so it would be convenient to generate one or more large sets of Glasbey colors and distribute them in colorcet.

Here's an example of generating 257 colors, which took about 5 minutes to run:

pip install colorspacious
git clone https://github.com/taketwo/glasbey.git
cd glasbey
python glasbey.py --view 257 --format float output.csv

You can click on the image to see it in more detail. The resulting list of color triples can easily be added to colorcet:

1.000000,1.000000,1.000000
0.000000,0.000000,0.000000
0.843137,0.000000,0.000000
0.549020,0.235294,1.000000
...
0.462745,0.160784,0.286275
0.741176,0.898039,0.000000

[jbednar]

Questions:

1. How many colors should we have? I chose 257 above so that when I took white off the front it would still have 256. Maybe it should be 258, so that I can take both white and black off the front?

2. Should any colors be omitted? E.g. shades of gray, or nearly gray?

3. The software allows starting with a base set of colors and extending it with additional ones. Should we start with a base set (e.g. Colorbrewer's Set1) so that the first few colors will match existing cycles, and other colors just extend it?

[jlstevens]

All good questions. My initial feedback is that even if we omit white and black, there are a good number of colors in there that are nearly white and nearly black that I doubt it would make a difference.

[jlstevens]

I suppose that answers point 2. If you are suggesting we remove colors that are close to anywhere on the grayscale colormap, then I think that is a good idea (for some definition of 'close').

[jbednar]

The code includes an option --no-black: avoid black and similar colors, which yields:

There's no --no-white option; not sure if that means it's avoiding white already. There are a few colors that do look close to white in the original set, including a light pink and a light beige.

Presumably we could filter the list returned ourselves to remove such colors, e.g. by omitting anything with average of the RGB channels below 0.1 and above 0.9? The set as a whole would then no longer truly be Glasbey colors, in that they wouldn't all be as distinct as they could be given the preceding colors, but it seems safer to remove colors than to add them.

[jbednar]

Or, to remove gray shades, we could filter out those where the R, G, and B channels are all within 5% of each other? Presumably I'd generate a larger set (e.g. 300), then filter out the grays, and take the first 256. As far as I understand the algorithm, the choices are made sequentially left to right, so it's ok to chop off colors at the end.

[jbednar]

Actually, I'm not sure about that -- the paper describes both a sequential algorithm and one for making a set of a certain size. Here I think we want the sequential algorithm, so that if a given plot only needs 32 colors, the first 32 colors in the map are maximally distinct from each other. I'll have to check the Python implementation to see which approach they are using.

[jbednar]

Ok, it is indeed using an iterative approach, which is clear in the code and verified by creating 25-color and 257-color palettes with the same options and visually ensuring that they are the same for the first 25 colors:

So we can indeed just generate 400 colors and take the top 256 after filtering, while acknowledging that the resulting set is not strictly Glasbey but a subset of them. So, do we want to remove any (types of) colors? Do we want to force the first few colors to be the same as any existing cycle?

[jbednar]

To get rid of grays, I implemented filtering out colors with a small range between the max and the min of the R,G,B values, and it seems to work well.

Filtering those with max range of 0.05 on a scale 0 to 1 from the first 400 eliminates only clearly monochrome colors (with the first image showing the colors filtered out, and the second showing those that remain):

Filtering up to 0.1 removes some clearly non-monochromatic colors, but they are still quite unsaturated:

By 0.15 there are definitely clearly visible non-monochromatic colors:

0.2 looks about the same as 0.15:

0.3 filters out nearly as many colors as it leaves behind:

(click on any of those images to enlarge them)

How about 0.1?

[jbednar]

Jupyter notebook for the above filtering, for reference (run in the glasbey directory from the above Python implementation):
Filter_Grays.zip

[jbednar]

I also tried what seems like a more principled approach, more truly following Glasbey, by removing grayish colors before colors are chosen. Specifically, I took all the available colors, transformed them to a color space where chroma is expressed explicitly, and filtered out all those whose chroma was less than a min_chroma value, modifying glasbey.py:

python glasbey.py --view 257 --min_chroma=10 --format float output.csv

In this approach it's not possible to see the specific greyish colors that are filtered out of the final cycle as above, since they are never generated in the cycle at all, but to get an idea of the sensitivity we can set the max chroma to different values and see the range of colors returned, then see the corresponding set of colors generated with that min chroma (again click to see the full image):

max_chroma=2

min_chroma=2

max_chroma=5

min_chroma=5

max_chroma=10

min_chroma=10

max_chroma=20

min_chroma=20

max_chroma=50

min_chroma=50

In each of these, ignore the first color being white in the final cycle -- that color is inserted by glasbey.py as the starting point, so it would need to be removed from the cycle explicitly. The second value is also blackish, though, all the way up to a min_chroma of 20, so it's surprising that such a color isn't being rejected. My guess is that its that black colors have a relatively high chroma value but that I can't perceive the chroma very well at that brightness (at least on this monitor), so I'll need to try also eliminating colors near black explicitly (which is already an option).

[jbednar]

Using --no-black gives more intuitive results:

min_chroma=2

min-chroma=5

min_chroma=10

min_chroma=20

Just as for --min_chroma, --no-black removes blackish colors before selecting colors for the cycle, preserving the maximally distinct property of each subsequent color (from the remaining universe of non-filtered colors (no grays, no blacks). Maybe min_chroma=10?

But note that the first color chosen when starting from white is always a very dark color (understandably, given the algorithm), so maybe we should start from black and white being the first two colors in the palette, generating the rest, then omitting black and white?

[jbednar]

Here's the result from doing the same as in the previous comment but initializing with a black and white palette, to make the subsequent colors avoid both of those:

python glasbey.py --base-palette=palettes/bw.txt --no-black --view 258 --min_chroma=2 --format float output.csv

min_chroma=2

min_chroma=5

min_chroma=10

min_chroma=20

Of course, the first two colors would then be deleted from the cycle, which is why there are 258 colors here.

[philippjfr]

Filtering before generating the palette definitely seems like the right approach. I can't quite see the benefit starting with black and white and then omitting them? Are you just trying to avoid the first non BW color being too dark? Visually the last two sets seem about as good as each other.

[philippjfr]

The software allows starting with a base set of colors and extending it with additional ones. Should we start with a base set (e.g. Colorbrewer's Set1) so that the first few colors will match existing cycles, and other colors just extend it?

This sounds like it would be worth a try, should maybe also try seeding it with the pandas default palette.

[jbednar]

The code requires some starting set of colors, which by default is just white. If I use the default, the second color is always very dark, which doesn't make for a very recognizable first color to choose for plotting. But if I add both white and black as the starting colors, the initial colors in the subsequent sequence seem more useful, in that they are actually nameable colors.

What about initializing with a given color sequence, like Set1? In practice for how we use color cycles, would that make such a palette more useful, in that it can be substituted for the default cycles more easily, extending them while preserving their current behavior?

[jsignell]

Ok so we discussed merging this without initalizing on Cateory6 or Category20 since those differ between bokeh and holoviews. Perhaps down the road, CategoryN will depend on this.

[jbednar]

I think we should probably just make all four versions:

1. Glasbey (initial values white and black, which are then discarded)
2. GlasbeyCategory20 (initial values white and black (I think!) plus Category20's colors, then discarding only white and black)
3. GlasbeyCategory6 (same as 20 but with 6 instead of 20)
4. GlasbeyPandas (same first colors as Pandas .plot())

That's assuming the 6 colors in Category6 don't match the first 6 of Category20? I'm not actually sure what Category6 is from -- is that the default in HoloViews?

[flutefreak7]

I got exposed to Glasbey colors in Julia's dashboard talk on YouTube and I was excited reading through this that a large palette might become importable soon without having to generate it. I was a little surprised between this and palettable no one had done this. Any progress on this?

I'm in aerospace engineering and we often have large historical datasets to show how the new data compares to the old and identifying 1 line among many is difficult with default palettes that only have 7-10 colors. Even Excel rotates through shades of it's theme colors to create dozens of unique colors so to me this is a place where all the python plotting libraries fall short when you want to plot more than 10 categorical lines.

This discussion is a bit lacking without some context on the myriad similar efforts besides the Glasbey library: https://graphicdesign.stackexchange.com/a/3815

There might even be some other large categorical color groups (like London subway, Kelly's 22, the color alphabet, etc) that deserve a spot in colorcet or palettable.

I like that the large palette you're generating is additive so that if I add a 16th color to my 15 color plot I don't have to reshuffle the colors... and that way in a report or notebook if I keep using the same palette, it becomes familiar to the reader. That weird experiment, run No. 13, is always that seafoam green color in all the plots, for example.

[jbednar]

No progress yet, but @jsignell has promised to take it on. We'd like to start being able to use these too! Thanks for the links; those do provide good background.

[flutefreak7]

Just came across this in R: https://cran.r-project.org/web/packages/pals/vignettes/pals_examples.html

It appears to be a more comprehensive collection of palettes than I've found anywhere else and an exhaustive collection of every palette evaluation tool. I'm not an R guy, but it is certainly impressive.

Also saw some interesting colormaps in the recent Plotly Express announcement called light24 and dark24 that appear to also come from R and are derived from a similar (inferior I think) methodology to Glasbey but with the goal of producing both a light and dark palette (so, like Brewer's Set1 and Set2 kinda).

https://medium.com/@plotlygraphs/introducing-plotly-express-808df010143d

https://rdrr.io/cran/Polychrome/man/custompal.html

[jbednar]

Pal is a useful reference, thanks! I'd be happy to see some of those colormap-evaluation tools ported into colorcet, such as pal.safe() (probably just requires adapting the code in https://github.com/matplotlib/viscm), pal.csf(), and maybe pal.test().

As far as which colormaps to include, for colorcet I've deliberately not tried to be comprehensive, because that would mean including a lot of colormaps that I don't think anyone should use. The continuous perceptually uniform palettes from cmocean, recent matplotlib, and those already in colorcet are all good; just about every other continuous colormap out there is a recipe for making misleading plots, and I don't want to facilitate that! :-)

As for categorical colors, colorbrewer is great for small numbers of colors, though probably nearly any other collection of fewer than 10 colors will be reasonable as long as they are suitably distinguishable. There seem to be a lot of people proposing various sets of around 25 colors, mostly without much principle, so I'd rather avoid most of those to keep things in colorcet more focused. That said, the light24 and dark24 ones there do seem useful so that they will suitably contrast with the background colors of the plot. I don't know of any large (>50) set of colors besides Glasbey that are usable for categories. We could consider making 256-color light and dark Glasbey sets by trimming the color space before picking colors, as we already to do avoid gray...

[jbednar]

Ok, we did all that --- colorcet now has various 256-color Glasbey sets that all start from an initial color set that includes black and white, optionally adding additional colors to the initial set:

• glasbey_bw, (Full set)
• glasbey_category10: Starting from Bokeh's Category10 colormap
• glasbey_hv: Starting from HoloViews default 10-color cycle

There are also additional sets that all start from black,white but trim out portions of the color space before running the algorithm:

• glasbey, glasbey_bw_minc_20: Filtering out grays
• glasbey_light, glasbey_bw_minc_20_minl_30: Filtering out grays and low-brightness colors
• glasbey_dark, glasbey_bw_minc_20_maxl_70: Filtering out grays and high-brightness colors
• glasbey_warm, glasbey_bw_minc_20_hue_330_100: Filtering out grays and all but the warm colors
• glasbey_cool, glasbey_bw_minc_20_hue_150_280: Filtering out grays and all but the cool colors