Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for dim transforms to set style options #2152

Merged
merged 117 commits into from Nov 22, 2018
Merged

Add support for dim transforms to set style options #2152

merged 117 commits into from Nov 22, 2018

Conversation

@philippjfr
Copy link
Member

@philippjfr philippjfr commented Nov 25, 2017

As discussed in #768 this PR adds support for scaling various style options by values in the element. It adds an op object which allows defining deferred transforms on an element dimension implementing the standard mathematical operations and common NumPy operations.

You can see what is currently supported here.

As we had discussed in the issue we will also support tuple and string syntax so you only have to define an op to define complex transforms more easily.

Here are some example:

scatter = hv.Scatter(np.arange(11))
o = (op('y')-op('y').min())/op('y').max()
print(o)
o.eval(scatter)
truediv(sub('y', amin('y')), amax('y'))
array([ 0. ,  0.1,  0.2,  0.3,  0.4,  0.5,  0.6,  0.7,  0.8,  0.9,  1. ])
%%opts Points (alpha=op('z') size=np.log1p(op('z'))*20)
hv.Points(np.random.rand(100, 3), vdims=['z'])

bokeh_plot

%%opts Curve [show_legend=False width=500] (line_width=10*op('y').mean())
hv.NdOverlay({i: hv.Curve(np.random.rand(10)) for i in range(5)})

bokeh_plot

@jlstevens
Copy link
Contributor

@jlstevens jlstevens commented Dec 12, 2017

I'll be happy to see this supported though I do have one request...

I really want to see op used in the simplest cases, matching what we already support. One goal of op is to support more advanced control than has been possible so far (which your examples above demonstrate) but another goal should be to keep simple things looking simple.

I want to make sure the simplest uses of op 1. work as a good default in many cases 2. remains succinct and readable.

Edit: I see a few examples in the notebook you linked to, though a few more examples shown in the comments of this PR wouldn't go amiss!

@philippjfr
Copy link
Member Author

@philippjfr philippjfr commented Jan 4, 2018

I'd also like this PR to address #237.

@jlstevens
Copy link
Contributor

@jlstevens jlstevens commented Jan 4, 2018

How do you think op can help with #237? It doesn't seem obvious to me yet...

@philippjfr
Copy link
Member Author

@philippjfr philippjfr commented Jan 4, 2018

I've discussed this before but I was going to suggest an apply method (or similar) for elements, which would let you transform dimensions/columns using ops or simply replace it. I'll put together a fully fleshed out proposal at some point.

@jlstevens
Copy link
Contributor

@jlstevens jlstevens commented Jan 4, 2018

I remember discussing apply with you but I didn't remember the idea that it could make use of op. It seems like a reasonable idea but I don't see why that would need to be part of this PR...

@philippjfr
Copy link
Member Author

@philippjfr philippjfr commented Jan 4, 2018

It wouldn't of course, could be a followup PR. I just wanted to establish a link between ops and that issue, so I don't forget about it.

@philippjfr
Copy link
Member Author

@philippjfr philippjfr commented Jun 23, 2018

To track how complete this PR is I'm going to make a list of elements that are fully tested:

Elements tested

Charts

  • Scatter/Points
    • Matplotlib
    • Bokeh
  • Curve
    • Matplotlib
    • Bokeh
  • Histogram
    • Matplotlib
    • Bokeh
  • Bars
    • Matplotlib (not supported, plot implementation is a mess)
    • Bokeh
  • Spikes
    • Matplotlib
    • Bokeh
  • Labels
    • Matplotlib
    • Bokeh
  • VectorField
    • Matplotlib
    • Bokeh
  • ErrorBars
    • Matplotlib
    • Bokeh

Geometries

  • Path
    • Matplotlib
    • Bokeh
  • Contours
    • Matplotlib
    • Bokeh
  • Polygons
    • Matplotlib
    • Bokeh

Graphs

  • Graph
    • Bokeh
    • Matplotlib
  • Chord
    • Bokeh
    • Matplotlib
  • Sankey
    • Bokeh
    • Matplotlib
  • TriMesh
    • Matplotlib
    • Bokeh

Statistics

  • BoxWhisker
    • Bokeh
    • Matplotlib (not supported)
  • Violin
    • Bokeh
    • Matplotlib (not supported)
  • HexTiles
    • Bokeh
    • Matplotlib (not supported)

Not supported (i.e. no vectorized glyphs)

  • Raster
  • QuadMesh
  • Image
  • RGB
  • HSV
  • HeatMap
  • Area
  • Spread
  • Distribution
  • Bivariate
  • Text
  • HLine
  • VLine
  • Spline

Note that when I something is not supported that means that this property cannot be set as a vectorized property. Therefore to support these cases we would have to plot multiple separate artists/glyphs, which won't be handled in the first iteration of this feature.

@philippjfr
Copy link
Member Author

@philippjfr philippjfr commented Jun 23, 2018

This notebook also tracks what is and isn't supported yet:

https://anaconda.org/philippjfr/op_transforms/notebook

@philippjfr
Copy link
Member Author

@philippjfr philippjfr commented Nov 21, 2018

So I think we are now in a place where this can be merged, but there are still a few outstanding items which will need to happen in later PRs:

  • Matplotlib BarPlot needs rewriting from scratch, I'm not touching the current implementation with a 10 foot barge pole
  • SankeyPlot does not yet support style mapping, this will require some work because the styles need to be carried through the operation
  • I need to finish writing a style mapping tutorial and or integrate the material in some existing tutorial
  • Eventually we want to support applying ops directly to elements to either transform existing dimension values or add new ones
  • Write more unit tests for plot updates

Some other features this PR introduces:

  • The normalization system now handles normalizing Graph edge start/end values along the node indices which is important for consistent coloration between nodes and edges
  • The normalization system now handles computing discrete factors across plots

Another idea I've had, is that we could allow specifying a transform(s) directly on a Dimension object so when you query the dimension on an element the transform is applied. This could be very powerful as it could support a huge range of transformations:

  • Unit transforms, e.g. when querying the values would be transformed to some fixed unit
  • Coordinate system transforms
  • Storing regularly spaced log transformed values from datashader on Image coordinates (solving #2195)
  • Jitter of axis locations
  • Lots more I can't think of right now
@jbednar
Copy link
Member

@jbednar jbednar commented Nov 21, 2018

Sounds great!

@philippjfr
Copy link
Member Author

@philippjfr philippjfr commented Nov 22, 2018

Tests should pass shortly. Let's get this merged so we can get a few days of testing in before a release.

@@ -50,7 +50,7 @@
"metadata": {},
"outputs": [],
"source": [
"violin.options(height=500, width=900)"
"violin.options(height=500, width=900, violin_fill_color=('Year', str), cmap='Set1')"

This comment has been minimized.

@jlstevens

jlstevens Nov 22, 2018
Contributor

I think this is a stray instance of the tuple format...

@@ -53,7 +53,7 @@
"metadata": {},
"outputs": [],
"source": [
"boxwhisker.options(show_legend=False, width=400)"
"boxwhisker.options(show_legend=False, width=600, box_fill_color=('origin', str), cmap='Set1')"

This comment has been minimized.

@jlstevens

jlstevens Nov 22, 2018
Contributor

And another tuple...

@jlstevens
Copy link
Contributor

@jlstevens jlstevens commented Nov 22, 2018

I'm happy to merge but I think I spotted a few stray instances of the tuple format above.

@jlstevens
Copy link
Contributor

@jlstevens jlstevens commented Nov 22, 2018

Tests are now green.

I'm very happy to merge this PR as I think it will greatly simplify things throughout while exposing a more powerful way of working. It will require a fair bit of testing and of course we may well have various tweaks to make. For the time being, I'll finally go ahead and merge...nearly a year since this PR was first opened!

@jlstevens jlstevens merged commit 9fd89a6 into master Nov 22, 2018
4 checks passed
4 checks passed
continuous-integration/travis-ci/pr The Travis CI build passed
Details
continuous-integration/travis-ci/push The Travis CI build passed
Details
coverage/coveralls Coverage increased (+0.9%) to 89.657%
Details
@philippjfr
s3-reference-data-cache Test data is cached.
Details
@philippjfr philippjfr deleted the ops branch Dec 13, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

3 participants