Color blind friendly color palettes

[mtennekes]

A follow-up on #566 and #593 In search for qualitative good color palettes, color blind friendliness is crucial.

To get started, I calculated three quality scores for diverging color palettes that are currently included in tmap4 (from grDevices, pals, and rcartocolor), and plotted World maps.

The quality scores I computed are:

1. inter_wing_dist minimal distance between from one wing (any color) to the other wing (any color). The idea is that the wings (or arms as mentioned by @zeilies Color palettes: native support for the pals package? #566 (comment)) should be as distinguishable as possible.
   Let a step be the distance from one color to a neighboring color, then:
2. min_step is the minimal step
3. step_indicator represents the homogeneity of the steps, more precisely, it represents how much the smallest or largest step deviates from the mean step: max(abs(step - mean(step))) / mean(step).

These three quality indices are computed for all three color vision deficiency types: per quality indicator, the worst score is returned. I don't care about normal color vision, because the quality scores should be better by definition. Note: I took the distance values for colorblindcheck::palette_dist() for granted. I haven't studied individual H, C, L levels yet.

See the reproducible script: sandbox/color_blind_check.R
I also computed a total score between 0 (bad) and 1 (good). See script how I did this.

According to this score, hcl palette vik (grDevices::hcl.colors(7, palette = "vik")) is the best one.

output of this script, maps for the top 30 palettes

However, there are two additional nice-to-haves:

1. The middle color should ideally not be white, since this may conflict with the background map (as mentioned by @Robinlovelace New tmap v4 defaults #593 (comment))
2. There should be a suitable color for missing values. Of course they could simply be omitted, but ideally there should be a separate color (e.g. light grey).

Which ones do you like?

My approach is to use this kind of analysis 1) to filter out color-blind-unfriendly palettes from the tmap color palette catalogue, and 2) to use one as a new default.

If you have any further ideas @Nowosad and @zeileis let me know.

[Robinlovelace]

Not a fan of white middle colour that's for sure... Is there a list of options that can be pasted in the chat below?

[zeileis]

Thanks, Martijn @mtennekes for this follow-up. I will look at this in more detail as soon as I can. Just a couple of quick comments now already:

• I think diverging is the more commonly-used term for these palettes while bivariate is used for these types of palettes: https://en.wikipedia.org/wiki/Multivariate_map.
• The "spectral" diverging palettes with yellow (typically) as the neutral color are typically perceived as warmer and also as more "interesting" because there is just more change in hues - as reflected in the comment by Robin @Robinlovelace . However, due to the many different hues and due to relatively high chroma throughout the palette, it is not so easy to distinguish postive-neutral-negative as it is in a palette that only uses two hues and light gray as the neutral color.
• Therefore, I would use such "spectral" diverging palettes when it's not so clear-cut whether I need a sequential or a diverging palette, e.g., for visualizing temperature over the year with the annual average used as the "neutral" point in the middle.
• But if I have diverging data where the two arms/wings are really fundamentally opposed, then a spectral diverging palette is typically sub-optimal. For example, climate warming vs. cooling, vegetation greening vs. browning, agreement vs. disagreement in polls etc. For these a palette with only two hues and a neutral gray in the middle is easier to read because the positive-neutral-negative decision is much more instantaneous.
• For the latter type of diverging palette I think it's also not necessary to maximize the distance of the colors in the two arms, I think. It's just important that the two hues remain sufficiently clearly distinct under color vision deficiencies.
• Finally, another advantage of these palettes is that you can switch between very light gray as the neutral color when you're using a white background and very dark gray as the neutral color when you're in dark mode with an (almost) black background.
• So in short: As the default for the diverging palette I would use something with just two hues and light gray as the neutral color as I think this is geared towards the most important application case (or the application case that is most distinct from sequential).
• The palettes that fulfill this property and remain most balanced in terms of chroma under all color vision deficiencies are Purple-Green and Vik. I like the former better because it is closer to the commonly used red/negative vs. green/positive analog.

What I still have to check in detail are the scores that you used to rank the palettes and how I would use these. But this will probably need a bit more time

[mtennekes]

Not a fan of white middle colour that's for sure... Is there a list of options that can be pasted in the chat below?

Sure:

cat(paste(.tmap_pals$fullname[.tmap_pals$type=="div"], collapse=", "))
hcl__bluered, hcl__bluered2, hcl__bluered3, hcl__redgreen, hcl__purplegreen, hcl__purplebrown, hcl__greenbrown, hcl__blueyellow2, hcl__blueyellow3, hcl__greenorange, hcl__cyanmagenta, hcl__tropic, hcl__broc, hcl__cork, hcl__vik, hcl__berlin, hcl__lisbon, hcl__tofino, hcl__armyrose, hcl__earth, hcl__fall, hcl__geyser, hcl__tealrose, hcl__temps, hcl__puor, hcl__rdbu, hcl__rdgy, hcl__piyg, hcl__prgn, hcl__brbg, hcl__rdylbu, hcl__rdylgn, hcl__spectral, hcl__zissou1, hcl__cividis, hcl__roma, misc__coolwarm, ocean__ocean.balance, ocean__ocean.curl, ocean__ocean.delta, brewer__brewer.brbg, brewer__brewer.piyg, brewer__brewer.prgn, brewer__brewer.puor, brewer__brewer.rdbu, brewer__brewer.rdgy, brewer__brewer.rdylbu, brewer__brewer.rdylgn, brewer__brewer.spectral, kovesi__kovesi.diverging_isoluminant_cjm_75_c23, kovesi__kovesi.diverging_isoluminant_cjm_75_c24, kovesi__kovesi.diverging_isoluminant_cjo_70_c25, kovesi__kovesi.diverging_linear_bjr_30_55_c53, kovesi__kovesi.diverging_linear_bjy_30_90_c45, kovesi__kovesi.diverging_rainbow_bgymr_45_85_c67, kovesi__kovesi.diverging_bkr_55_10_c35, kovesi__kovesi.diverging_bky_60_10_c30, kovesi__kovesi.diverging_bwr_40_95_c42, kovesi__kovesi.diverging_bwr_55_98_c37, kovesi__kovesi.diverging_cwm_80_100_c22, kovesi__kovesi.diverging_gkr_60_10_c40, kovesi__kovesi.diverging_gwr_55_95_c38, kovesi__kovesi.diverging_gwv_55_95_c39, carto__ArmyRose, carto__Earth, carto__Fall, carto__Geyser, carto__TealRose, carto__Temps, carto__Tropic

See also this post #566 (comment)

[mtennekes]

Yes, I meant diverging and not bivariate. Typo:-)

I am struggling the most with the middle color in combination with the color for missing values (and optionally a greyscale background map). Personally, I also don't like white as middle color, but dislike black even worse.

There are a few palettes that use another hue for middle color, not only "RdYlBu" (yellow), but also "Roma" (turquoise). I agree with @zeileis that another hue is tricky to use, although I still find "RdYlBu" okay.

There are only a few palettes that use light grey (but not too light): e.g. "kovesi.diverging_bwr_40_95_c42" and "coolwarm". I find the middle grey of both "Vik" and "purple-green" too bright, i.e. too close to white. Using a darker grey while keeping the other colors the same is also tricky I guess(?)

From a practical point of view, using a grey middle color using a greyscale background map (e.g. StamenTonerLite or CartoDBPositron or Esri.WorldGreyCanvas (https://leaflet-extras.github.io/leaflet-providers/preview/)) does not work well. For those applications a hued middle color is better. I am not looking for a perfect diverging palette: we could recommend "RdYlBu" to be used with a greyscale background map, and "purple-green" otherwise.

Still a color for missing values is needed. We could quantitatively determine the best color by computing the color in the (HCL) color space with the largest distance to any color in the (given) palette. This should be computed for each of the different CVD types. I am very curious what comes out.

I agree that maximizing the distance in each arm is less important than discriminating the two arms from each other. Please feel free to experiment with the quality indicators. Mine are just initial ideas, but far from perfect. As target we could aim for quality functions to include in the colorblindcheck package, although my own priority is tmap.

[mtennekes]

Tagging @mputs who knows a lot more about visual perception than I do. Marco: this is the link with 30 diverging color palettes with color vision deficiency simulations: https://github.com/r-tmap/tmap/files/7978336/biv_plots.zip

[zeileis]

Martijn @mtennekes could you expand on why it is important that the neutral color needs to be very different from the background color? Do you have a pair of examples that illustrates why this can be important?

I'm asking because usually I would argue that the neutral color should almost blend into the background. For me, diverging palettes are about bringing out whether something is "interesting" in two directions "interesting & positive" vs. "interesting & negative". So the neutral observations in the center are "uninteresting" and hence it's ok if these disappear into the background. But maybe I'm thinking too much of heatmaps than of actual maps. So an example would help me here.

As for the color for missing values: I guess this depends on how much you want these to stand out. Using white is often a good compromise between being clearly visible while not being too distracting at the same time. But on a black/dark background it may still be too heavy.

[mtennekes]

Thanks for bringing this up, Achim @zeileis. I agree that the extremes of a diverging palette are more interesting, but it is also important to know where features are that have a neutral color.

Example from an older project at Statistics Netherlands in which we estimated the present population per municipality:

When light grey was used instead of yellow, the shape of the Netherlands would have been less clear.

Neutral color are even more important when plotting transport/road network data, e.g. traffic intensities. As a user I would like to know which roads are included in the analysis, which would be less clear if the neutral color blends in with the background map. Perhaps Robin @Robinlovelace has some examples?

[Robinlovelace]

I do have examples, will dig them out...

[Robinlovelace]

Thanks for asking, I've found this problem many times and I think it's rather specific to maps, on which data must be displayed over a variable (grey/white, sometimes coloured) basemap. This is a good example that cause me to make a custom colourscheme with {colorspace}.

[Robinlovelace]

[zeileis]

Thanks for sharing these examples, this indeed helps to understand your points.

In Robin's two maps I agree that it is important to have colors with enough chroma to make the shading visible on the map at all. In this particular application, though, it's not so clear that this needs to be a diverging rather than a sequential palette. But I can easily imagine a similar map with a quantity that really needs a diverging palette - and then you need something with enough chroma throughout.

In the map of the Netherlands that Martijn shared, my impression is that there is too much chroma. I have the feeling that I have to concentrate much more on where and how the patterns change.

[mtennekes]

Thanks Robin and Achim for your input, very helpful!

Based on these insights and examples we could already make some guidelines:

• When a background map is used, the chroma of the color palette should be sufficiently higher (but not too much) than that of the background map. The reasoning behind this is that chroma is a depth cue for visual perception: we perceive objects that are farther away as less saturated (more greyish). By choosing a palette that has just a bit higher chroma, we perceive this data layer as sitting on top of the baselayer. This could also explain why you had to concentrate more on my example Achim: when the difference of levels of chroma between two map layers is too large, it creates an unnatural situation which confuses the visual system (does this make sense @mputs?). For this reason I recommend to use greyscale basemaps, or if that is too extreme, to partly desaturate basemaps. (This is already possible in tmap's "plot" mode; maybe it can also be done in leaflet via some workaround?).
• When there is no clear-cut between the two sides, a spectral palette is generally recommended, because it has the advantage that colors can be easier distinguished from each other. However, I'm not sure how people with color vision deficiency will perceive those (have to test that).
• When missing values need to be colored, white or very light grey will often work well (at least on a white/light-greyscale background)

This discussion also holds for sequential palettes: for palettes with one hue, one question is whether the brightest color stands out enough. Since there is no middle color in these palettes, it is often worthwhile to use a spectral palette.

Let me know if you have anything to add.

[Nowosad]

All of the visuals from the script mentioned in the first post are shown below.

Color vision deficiency: none

Color vision deficiency: deutan

Color vision deficiency: protan

Color vision deficiency: tritan

[Nowosad]

Maps for the top 30 palettes (from 30th to the 1st place) mentioned in the first post.

blueyellow2

Earth

berlin

redgreen

piyg

brbg

kovesi diverging_linear_bjy_30_90_c45

ocena delta

roma

cork

brewer rdylbu

rdylbu

kovesi diverging_bkr_55_10_c35

rdbu

ocean balance

tofino

coolwarm

bluered3

kovesi diverging_bwr_55_98_c37

cividis

puor

kovesi diverging_bwr_40_95_c42

purlplebrown

broc

prgn

purplegreen

kovesi diverging_gwv_55_95_c39

lisbon

kovesi diverging_bky_60_10_c30

vik

[Nowosad]

Just a few comments:

• I agree with @zeileis that it would be best to have a color friendly but also intuitive color palette and thus, purple-green seems to be a better default compared to vik
• @mtennekes if you think you could find a way to create distinct (and not ugly) colors for missing values -- that would be great to see
• In overall -- thanks for a great discussion and I think that these examples could be helpful for other people (and projects) as well

[zeileis]

Very nice to see so much progress, thanks everybody for the follow-ups and improvements. Just some short additions to Martijn's guidelines:

• In addition to chroma being too high compared to the background, the reason why I had to concentrate was the spectral nature of the palette. I find it much easier to spot spatial patterns in a map when there are only two hues compared the more qualitative changes you get when shifting hues over a larger spectrum.
• What is true is that it is often easier with spectral palettes to match certain colors/cells to the precise numeric values in the legend. However, this is really not what I'm after, especially when the map changes across time every second or so in an animation. Then I really want to understand the dynamics in the spatial patterns.
• The same holds for sequential palettes. When I'm more concerned about bringing out patterns and/or extremes only, then a sequential palette with one hue (or a rather small hue range) is easier to process. The multi-hue palettes make it easier though to match individual values to the legend.

[mtennekes]

Color blind friendly scores for all palettes that are currently available in tmap4. Next step is to reduce the palettes to color blind friendly palettes only, and organize them better.

The scores I applied (third column) are:

• Categorical: min_dist the minimal distance between any two colors
• Sequential: min_step the minimal step distance between any two neighboring colors; max_step the maximal step distance. The higher min_step the better; for palettes with equal min_step scores, those with lowest max_step are preferable.
• Diverging: inter_wing_dist the minimal distance between any color in one wing with any color in the other wing. min_step (see sequential).
• Cyclic: same as sequential, but the first and last color are also neighbors.

The scores are applied to all types of cvd: the worst score is taken.

Below, palettes are sorted from best to worst according these scores. For categorical palettes, I made 5 selections: those with at least 6, 8, 10, 12, and 20 colors respectively.

Some findings:

• Categorical: the color-blind-friendly palettes (tol, kelly and okabe) score very well indeed. However, imho including black and/or white is cheating (ggplot also doesn't color its bars black). For consistency, black should either be included in all palettes or in none.
• Sequential: we should organize these palettes by: single-hue, "via"-hue (e.g. YlOrBr) and rainbow/spectral (which is sometimes classified as diverging).
• Diverging: do not use (hcl/brewer) Spectral! Although the scores are okay, the hues of the first color of the left wing and the first color of the right wing are misleaning for deutans and protans:

top: the palette for people with normal color vision, bottom: for deutans (protans is similar)

Normal color vision

Deutan

Protan

Tritan

[Robinlovelace]

Just to say: the palette options above look great, big improvement on the list you could see with tmaptools::palette_explorer().

[mtennekes]

Follow-up which closes this topic: https://github.com/mtennekes/cols4all
@zeileis @Robinlovelace @Nowosad : I have made you contributors of this new package, because of your helpful feedback and discussion.

[Robinlovelace]

Great to see this Martijn. As a minor follow-up, here's a colourscheme that seems to fit well.

sequential_hcl(n = 11, h = c(141, 6), c = c(70, NA, 86), l = c(73, 52), power = c(0.7, 1.9), register = )

[mtennekes]

Thanks Robin!

remotes::install_github("mtennekes/cols4all")
library(cols4all)
pals = list(cycling = sequential_hcl(n = 11, h = c(141, 6), c = c(70, NA, 86), l = c(73, 52), power = c(0.7, 1.9)))
c4a_series_add_as_is(pals, xNA = NA, types = "seq", series = "robin")
c4a_gui()

gives

So your palette is certainly a good starting point. It is well-balanced, and doesn't have intense (high chroma) colors. For 'protan' the colors are still a bit too similar. I have set the threshold value for "minimal step" to 5 (5+ means "color-blind-friendly"), and your palettes scores 2.