New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

adds two new cyclic color schemes #6254

Merged
merged 17 commits into from May 28, 2018

Conversation

Projects
None yet
@bastibe
Contributor

bastibe commented Mar 31, 2016

Matlotlib currently lacks a good, perceptually uniform color scheme for angular data. This adds "sunlight" and "twilight", two viscm-derived color schemes for angular data.

These color schemes go from white-blue-black-red-white (twilight) and black-red-white-blue-black (sunlight), and are perfectly suited for angular data where low values wrap around to high values. The color schemes print as black-white-black and white-black-white respectively, and work well with color blindness.

@mdboom mdboom added the Needs review label Mar 31, 2016

@bastibe

This comment has been minimized.

Contributor

bastibe commented Mar 31, 2016

Now this should pass the pep8 test.

@bastibe

This comment has been minimized.

Contributor

bastibe commented Mar 31, 2016

twilight

@bastibe

This comment has been minimized.

Contributor

bastibe commented Mar 31, 2016

sunlight

@@ -19,4 +19,5 @@
'brg', 'CMRmap', 'cubehelix',
'gnuplot', 'gnuplot2', 'gist_ncar',
'nipy_spectral', 'jet', 'rainbow',
'gist_rainbow', 'hsv', 'flag', 'prism'])]
'gist_rainbow', 'hsv', 'flag', 'prism']),
('Perceptually Uniform Cyclic', ['sunlight', 'twilight'])]

This comment has been minimized.

@madphysicist

madphysicist Mar 31, 2016

Contributor

This should probably go before Miscellaneous.

@@ -57,7 +57,8 @@
'brg', 'CMRmap', 'cubehelix',
'gnuplot', 'gnuplot2', 'gist_ncar',
'nipy_spectral', 'jet', 'rainbow',
'gist_rainbow', 'hsv', 'flag', 'prism'])]
'gist_rainbow', 'hsv', 'flag', 'prism']),
('Perceptually Uniform Cyclic', ['twilight', 'sunlight'])]

This comment has been minimized.

@madphysicist

madphysicist Mar 31, 2016

Contributor

Any reason the order was switched here from doc/users/plotting/colormaps/colormaps.py ?

This comment has been minimized.

@bastibe

bastibe Apr 1, 2016

Contributor

No reason at all, I'll immediately fix it.

@madphysicist

This comment has been minimized.

Contributor

madphysicist commented Mar 31, 2016

Besides the trivial comments I made, this is very neat.

@efiring efiring added this to the 2.0 (style change major release) milestone Mar 31, 2016

@efiring

This comment has been minimized.

Member

efiring commented Mar 31, 2016

This looks like a very useful addition. I wonder whether it is possible to smooth the behavior in the middle of the sunlight map, though.

@QuLogic

This comment has been minimized.

Member

QuLogic commented Mar 31, 2016

Can you produce an example that specifically demonstrates the cyclic nature of these maps?

@bastibe

This comment has been minimized.

Contributor

bastibe commented Apr 1, 2016

@efiring The non-smoothness is a result of the old version of viscm I used to create the colors. IIRC, that version had a bug in it's color space model, which leads to the non-smooth middle.
I still have the design files though, so this should be fixable. I'll get right on it.

@bastibe

This comment has been minimized.

Contributor

bastibe commented Apr 1, 2016

@QuLogic Here is an example of a special kind of phase spectrum I am working on, where high values (+75) are equivalent to low values (-75). I attach the same plot in viridis and sunlight:

viridis

sunlight

As you can see, the big wavy lines at the bottom cycle through the value range in the Y direction. This is immediately apparent in the sunlight plot, whereas the viridis plot looks more like homogenous horizontal mid-green bands bounded by chaotic yellow/blue boundaries.

Also notice how the background (top/left) looks mostly white, with some red and blue in sunlight, whereas the viridis plot makes it hard to judge whether there is more yellow or more blue.

These kinds of effects happen regularly with angular data, i.e. data where high values are synonymous with low values. Typically these will be angles that go from -pi to pi or from 0 to 2*pi, like the angle of complex numbers, or wind directions. In the above case, it is some derivative of an angle, which still wraps high values to low values.

@bastibe

This comment has been minimized.

Contributor

bastibe commented Apr 1, 2016

@efiring I updated the colormap to the latest color space model of viscm. Apparently, this also made it a bit more saturated. The latest commit changes the saturation back towards the original commit. For reference, I attached the updated viscm-visualizations again.

The short minimum of the perceptual deltas in the middle (and invisibly, the ends) is unavoidable due to the reversal of lightness direction I think.

twilight_viscm
sunlight_viscm

@bastibe

This comment has been minimized.

Contributor

bastibe commented Apr 1, 2016

I am happy with how this looks now. Feel free to merge if you like it, too!

In my area of speech signal processing, phase spectra are pretty common, but judging from publications in the area, most people use very inappropriate color maps for it. Sunlight/Twilight is a perfect fit for this kind of data, and one that is not present in i.e. Matlab.

@tacaswell

This comment has been minimized.

Member

tacaswell commented Apr 1, 2016

@efiring

This comment has been minimized.

Member

efiring commented Apr 1, 2016

I really like these two new maps, but there is still a discontinuity in the middle, so that sunlight makes a white stripe and twilight makes a black stripe. This is immediately evident by eye in the "colormap in its glory" picture, and it is only confirmed by the downward spike in the perceptual deltas plot. It looks like there is still some tiny adjustment that could make a big visual difference. Maybe @njsmith will see right away how to do it.

@WeatherGod

This comment has been minimized.

Member

WeatherGod commented Apr 1, 2016

My bet? viscm probably repeating the same colors when it is "reflecting"
the two sides of the colormap, rather than only using the end values once

On Fri, Apr 1, 2016 at 3:35 PM, Eric Firing notifications@github.com
wrote:

I really like these two new maps, but there is still a discontinuity in
the middle, so that sunlight makes a white stripe and twilight makes a
black stripe. This is immediately evident by eye in the "colormap in its
glory" picture, and it is only confirmed by the downward spike in the
perceptual deltas plot. It looks like there is still some tiny adjustment
that could make a big visual difference. Maybe @njsmith
https://github.com/njsmith will see right away how to do it.


You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub
#6254 (comment)

@bastibe

This comment has been minimized.

Contributor

bastibe commented Apr 2, 2016

This is constructed from two concatenated viscm colormaps, which each start and stop at the same points. One goes off-black, red, off-white, the other goes off-black, blue, off-white. By making them start and stop at the same point, and choosing similar-saturation paths through the color space, the combined colormap appears nice and circular.

This is not an automated process, though. I hand-designed the two blue/red colormaps and hand-concatenated them, omitting the repeated endpoints. I just realized that I forgot about one of the end-points, and corrected this error. Thank you @WeatherGod and @efiring for pointing that out.

However, this leaves discontinuities at the points of concatenation, where the perceptual deltas get smaller for one color value. I experimented with the discontinuities by leaving out larger areas around the end points, but that does not make the discontinuity go away. I therefore conclude that the discontinuities exist because of the reversal in lightness, and are inevitable in a cyclic colormap.

Also, this colormap consists of 510 points, only two of which are not perceptually uniform (the points of concatenation). Two points out of 510 clearly are too narrow to cause visible black/white stripes as mentioned by @efiring.

@njsmith

This comment has been minimized.

njsmith commented Apr 2, 2016

A few scattered thoughts:

I think matplotlib probably needs to come up with some general guidelines for how they want to handle adding new colormaps -- in the limit you get in a situation where you have hundreds of random colormaps used by one person each, each with a more or less clever name. No idea where this submission would fall WRT that hypothetical policy, but it might be a good time to stop and discuss that? Or even possibly declare a moratorium on adding new colormaps until some guidelines can be formulated...

The sharp band in the white/black region is indeed a inevitable consequence of the sharp switch in the lightness gradient -- see this paper by Peter Kovesi for some discussion. He suggests applying a bit of smoothing to the lightness gradient to reduce the perceptual artifact.

@jerryz123 has a branch of viscm that would actually make it much easier to create this particular colormap -- what it actually adds is the ability to make diverging maps, not cyclic maps, but this particular cyclic map is basically a diverging colormap anyway :-). We've thought some about supporting cyclic colormaps directly, but it's a bit tricky since in general you need to be able to draw an arbitrary 3-dimensional curve (the Kovesi paper again has some good examples of the paths taken through perceptual space by good cyclic colormaps).

The difference between twilight and sunlight is just a phase shift, right? If this is intended as a cyclic map then in general I feel like a better solution to wanting to adjust the colormap phase is to provide tools in matplotlib to do that in general, rather than duplicating specific colormaps for different phase angles...

Peter Kovesi provides a better test image for cyclic colormaps: http://peterkovesi.com/projects/colourmaps/colourmaptestimage.html

He also provides a nice set of colormaps for that matter, including some cyclic ones: peterkovesi.com/projects/colourmaps/

I guess if I'm going to keep taking his name in vain then I should CC @peterkovesi :-)

@bastibe

This comment has been minimized.

Contributor

bastibe commented Apr 4, 2016

Thank you @njsmith for this very interesting response. There are two things I want to add:

First, in my field of research, a color map should be printable in black and white, which is why I chose two dark-color-bright diverging colormaps as a source for sunlight. Most of the cyclic color maps in the referenced paper are not easily printable in black and white.

Second, as mentioned in the paper, it is often desirable to have a clear reference point in angular data. For sunlight, these are the points of lightness reversal. Again, I find this very useful for my area of research, where a phase angle of 0 (and 180) has some significance. This point may be somewhat idiosyncratic for my particular application, though.

It would probably be useful to tag color maps in general with a set of properties such as

  • type (linear, diverging, cyclic)
  • perceptual uniformity
  • printable in black and white
  • colorblindness-friendly

It might be sufficient to provide this information in colormaps_reference.py.

It might also prove useful to have some methods for manipulating arbitrary color maps, such as

  • reversing
  • rotating (cyclic only)
  • perceptual smoothing (equal perceptual delta)
  • subsampling to a number of discrete colors
@bastibe

This comment has been minimized.

Contributor

bastibe commented Apr 6, 2016

Any news on whether this pull request needs more work, or whether it can be merged?

@njsmith

This comment has been minimized.

njsmith commented Apr 6, 2016

@bastibe: I'm not a matplotlib dev, but my (possibly wildly inaccurate) guess would be that this may need to sit for a while before any decision gets made, because of the judgement call issue in matplotlib having some desire not to end up with a bazillion random colormaps, but not yet having clear criteria for how to make that decision. Plus in general the devs are very busy trying to get out the delayed 2.0 release and other things like that.

@efiring efiring modified the milestones: 2.1 (next point release), 2.0 (style change major release) May 2, 2016

@efiring

This comment has been minimized.

Member

efiring commented May 2, 2016

@bastibe, I don't want to discourage you--I like what you are proposing here--but I also don't want this to hold up 2.0. We do need to sort out our strategy for dealing with new colormaps; maybe we can do this right after the 2.0 release, coming soon, we all hope.

@bastibe

This comment has been minimized.

Contributor

bastibe commented May 11, 2016

I implemented the lightness smoothing, as per the Kovesi paper mentioned by @njsmith. Here are the new test images:

sunlight
twilight

@bastibe

This comment has been minimized.

Contributor

bastibe commented May 11, 2016

For reference, here's the code that does the smoothing:

# original color data is in cm_data
import scipy.signal as sig
import numpy as np
from colorspacious import cspace_convert
# convert to CIELab color space:
cie_data = cspace_convert(cm_data, 'sRGB1', 'CIELab')
lightness = cie_data[:,0]
# lengthen to allow smoothing of both ends:
extended_lightness = np.concatenate([lightness[-100:],
                                     lightness[:],
                                     lightness[:100]])
# smooth with a gaussian window:
smoothing_sigma = 15 
smoothing_window = sig.gaussian(smoothing_sigma*8, smoothing_sigma)
smoothing_window /= np.sum(smoothing_window)
smooth_lightness = sig.filtfilt(smoothing_window, [1], extended_lightness)
# cut back to original size:
cie_data[:,0] = smooth_lightness[100:-100]
smooth_data = cspace_convert(cie_data, 'CIELab', 'sRGB1')

Kovesi recommends sigma=5...7 for a 256-bin color map, thus I use 15 for a 511-bin color map. Note that I use filtfilt, so the filter is applied twice, but no delay is introduced. This adds even more smoothing. I experimented with different values, though, and less smoothing does not seem to remove the false feature.

@OceanWolf

This comment has been minimized.

Contributor

OceanWolf commented May 15, 2017

@bastibe This looks really interesting, have you tried using the test image that @njsmith suggested? I would like to see that.

Not confident about the need for a "clear reference point", and it sounds like you want two. I thought the whole point in angular data comes from the lack of any reference point, it should look cyclic and therefore no discontinuities.

What would happen with use a Normalize class to map data into the [0,1] colourmap space, i.e. the default Normalize class we just use:

def norm(data):
  return (data - min) / (max - min)  # in MPL, min, max user specified, or auto-calculated

but instead for cyclic data, why not just Normalize the data like this:

def norm(data, period, data_offset=0, angular_offset=9)
  return np.cos((data + data_offset) * 2*np.pi / period + angular_offset)

I presume if we were to do this with viscim it would show a cyclic wave on the perceptual uniformity graph rather than a straight line, but I wonder if the brain can adjust for that as we see no sudden spikes. I would guess that this takes the "smoothing" to the extreme.

Note I use pseudo code, MPL's code uses slightly more complicated structure (classes, etcetera), but it boils down to simple functions like this. You don't even need to deal with MPL's normalization code to do this, as you can normalise by hand. Just use these functions directly to modify your data.

@tacaswell tacaswell removed this from the 2.1 (next point release) milestone Sep 24, 2017

bastibe added some commits Apr 1, 2016

update twilight to fixed color space in viscm
viscm fixed a bug in it's color space model. This uses the same design
parameters of the original color map in the new color space.
fix typo.
Sorry about that.
reduced saturation to be more like the original
the update to viscm's newer color space model changed the saturation of
twilight. This changes the parameters to make them more like the original.
remove duplicate end-point of twilight
twilight is constructed from two viscm-generated colormaps, one red, one blue, both starting and stopping at the same points. When originally created, they thus contain double start- and end-points. While I did take out the repeated black point, I forgot to delete one of the end-points. This fixes that omission.
add lightness smoothing
as mentioned in the Kovesi paper

@bastibe bastibe force-pushed the bastibe:colormap-twilight branch from e59ee75 to 07d25f3 May 9, 2018

@bastibe

This comment has been minimized.

Contributor

bastibe commented May 9, 2018

@jklymak OK, I think I rebased everything, and added a what's new entry. Is this all to your liking? Do you want me to squash the commits for a cleaner merge? Is there anything else you would like me to do?

I moved the hsv colormap from "Miscellaneous" colormaps to "Wrapping" colormaps. I hope this is correct?

What remains is a discussion about naming. I don't have a strong opinion here. Proposed names:

  • Twilight/Sunlight (or possibly, twilight_r)
  • Midnight/Noon
  • rdbu_cyclic/_r (what does rdbu stand for?)

Technically, the second color map is not exactly reversed, but shifted. But I guess the _r suffix is clear enough anyway.

@jklymak

This comment has been minimized.

Contributor

jklymak commented May 9, 2018

This is great. At some point a squash would be good.

Re naming; minor and as the author of this I guess you should get some priority in choice (within reason):

  • RdBu is just red-blue the standard diverging colormap which this one looks to be pretty similar to, but with better wrap characteristics.
  • I meant the _r just to mean the reverse colormap, not "twilight". RdBu_cyclic_light and RdBu_cyclic_dark? I dunno - maybe more evocative names are fine.

Please be sure to ping if this isn't talked about more for a week or so!

@jklymak

Still not 100% on the names, but otherwise this is great.

@jklymak jklymak removed the Needs rebase label May 17, 2018

@bastibe

This comment has been minimized.

Contributor

bastibe commented May 18, 2018

Sorry for not writing anything, it has been a busy week.

I think my preference would be twilight and twilight_shifted. Initially, I thought that we could use twilight_r instead of twilight_shifted, but _r and _shifted actually fulfill different roles (reversing blue/red vs. reversing light/dark).

To make this clearer, I changed the color order of twilight_shifted from black-red-white-blue-black to black-blue-white-red-black, so that the color order is now equal to twilight. I hope that this makes the differences between the two more understandable.

Is this to your liking? If so, I'll squash it, and hand it over to you.

@@ -206,13 +231,13 @@ def plot_color_gradients(cmap_category, cmap_list, nrows):
# Number of colormap per subplot for particular cmap categories
_DSUBS = {'Perceptually Uniform Sequential': 5, 'Sequential': 6,
'Sequential (2)': 6, 'Diverging': 6, 'Qualitative': 4,
'Miscellaneous': 6}
'Sequential (2)': 6, 'Diverging': 6, 'Wrapping': 3,

This comment has been minimized.

@OceanWolf

OceanWolf May 18, 2018

Contributor

What do people think about the category name of Wrapping? I found this a bit of surprise to see this here given that this PR was about cyclic colour schemes, I would have thought that would have been a more suitable name.

This comment has been minimized.

@bastibe

bastibe May 18, 2018

Contributor

I have no opinion on this. Either name is fine for me. Wrapping was proposed somewhere in the discussion.

This comment has been minimized.

@jklymak

jklymak May 22, 2018

Contributor

I don't feel strongly either way - cyclic didn't quite mean anything to me, but if that is a more common term, by all means change it back...

This comment has been minimized.

@efiring

efiring May 23, 2018

Member

To me, "cyclic" is more evocative of the character and intended use.

This comment has been minimized.

@bastibe

bastibe May 23, 2018

Contributor

No problem, I'll change it back to cyclic.

This comment has been minimized.

@bastibe

bastibe May 23, 2018

Contributor

done.

@jklymak

This comment has been minimized.

Contributor

jklymak commented May 28, 2018

This is now named by the concensus naming. Can we get a second approval and merge? This is a good idea that is over two years old....

@dstansby dstansby merged commit 8c8e1b5 into matplotlib:master May 28, 2018

8 checks passed

ci/circleci: docs-python35 Your tests passed on CircleCI!
Details
ci/circleci: docs-python36 Your tests passed on CircleCI!
Details
codecov/patch 100% of diff hit (target 50%)
Details
codecov/project/library 70.05% (target 50%)
Details
codecov/project/tests 98.98% (+0.1%) compared to 1652ed1
Details
continuous-integration/appveyor/pr AppVeyor build succeeded
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
lgtm analysis: Python No alert changes
Details
@jklymak

This comment has been minimized.

Contributor

jklymak commented May 28, 2018

Thanks so much @bastibe So sorry for the slow review process!

@endolith

This comment has been minimized.

Contributor

endolith commented Aug 18, 2018

@bastibe

It would probably be useful to tag color maps in general with a set of properties such as ...

It might be sufficient to provide this information in colormaps_reference.py.

I tried to document some properties in pyplot.colormaps(), and left a note at the top of _cm.py to update that each time a colormap is added, but I don't know if people have been doing that...

Actual properties of the colormap objects would be nice, too. Holoviews has the ability to list all uniform sequential maps, for instance:

http://holoviews.org/user_guide/Colormaps.html

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment