Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for dim transforms to set style options #2152

Merged
merged 117 commits into from Nov 22, 2018

Conversation

Projects
None yet
3 participants
@philippjfr
Copy link
Contributor

philippjfr commented Nov 25, 2017

As discussed in #768 this PR adds support for scaling various style options by values in the element. It adds an op object which allows defining deferred transforms on an element dimension implementing the standard mathematical operations and common NumPy operations.

You can see what is currently supported here.

As we had discussed in the issue we will also support tuple and string syntax so you only have to define an op to define complex transforms more easily.

Here are some example:

scatter = hv.Scatter(np.arange(11))
o = (op('y')-op('y').min())/op('y').max()
print(o)
o.eval(scatter)
truediv(sub('y', amin('y')), amax('y'))
array([ 0. ,  0.1,  0.2,  0.3,  0.4,  0.5,  0.6,  0.7,  0.8,  0.9,  1. ])
%%opts Points (alpha=op('z') size=np.log1p(op('z'))*20)
hv.Points(np.random.rand(100, 3), vdims=['z'])

bokeh_plot

%%opts Curve [show_legend=False width=500] (line_width=10*op('y').mean())
hv.NdOverlay({i: hv.Curve(np.random.rand(10)) for i in range(5)})

bokeh_plot

@jlstevens

This comment has been minimized.

Copy link
Contributor

jlstevens commented Dec 12, 2017

I'll be happy to see this supported though I do have one request...

I really want to see op used in the simplest cases, matching what we already support. One goal of op is to support more advanced control than has been possible so far (which your examples above demonstrate) but another goal should be to keep simple things looking simple.

I want to make sure the simplest uses of op 1. work as a good default in many cases 2. remains succinct and readable.

Edit: I see a few examples in the notebook you linked to, though a few more examples shown in the comments of this PR wouldn't go amiss!

@philippjfr

This comment has been minimized.

Copy link
Contributor Author

philippjfr commented Jan 4, 2018

I'd also like this PR to address #237.

@jlstevens

This comment has been minimized.

Copy link
Contributor

jlstevens commented Jan 4, 2018

How do you think op can help with #237? It doesn't seem obvious to me yet...

@philippjfr

This comment has been minimized.

Copy link
Contributor Author

philippjfr commented Jan 4, 2018

I've discussed this before but I was going to suggest an apply method (or similar) for elements, which would let you transform dimensions/columns using ops or simply replace it. I'll put together a fully fleshed out proposal at some point.

@jlstevens

This comment has been minimized.

Copy link
Contributor

jlstevens commented Jan 4, 2018

I remember discussing apply with you but I didn't remember the idea that it could make use of op. It seems like a reasonable idea but I don't see why that would need to be part of this PR...

@philippjfr

This comment has been minimized.

Copy link
Contributor Author

philippjfr commented Jan 4, 2018

It wouldn't of course, could be a followup PR. I just wanted to establish a link between ops and that issue, so I don't forget about it.

@philippjfr

This comment has been minimized.

Copy link
Contributor Author

philippjfr commented Jun 23, 2018

To track how complete this PR is I'm going to make a list of elements that are fully tested:

Elements tested

Charts

  • Scatter/Points
    • Matplotlib
    • Bokeh
  • Curve
    • Matplotlib
    • Bokeh
  • Histogram
    • Matplotlib
    • Bokeh
  • Bars
    • Matplotlib (not supported, plot implementation is a mess)
    • Bokeh
  • Spikes
    • Matplotlib
    • Bokeh
  • Labels
    • Matplotlib
    • Bokeh
  • VectorField
    • Matplotlib
    • Bokeh
  • ErrorBars
    • Matplotlib
    • Bokeh

Geometries

  • Path
    • Matplotlib
    • Bokeh
  • Contours
    • Matplotlib
    • Bokeh
  • Polygons
    • Matplotlib
    • Bokeh

Graphs

  • Graph
    • Bokeh
    • Matplotlib
  • Chord
    • Bokeh
    • Matplotlib
  • Sankey
    • Bokeh
    • Matplotlib
  • TriMesh
    • Matplotlib
    • Bokeh

Statistics

  • BoxWhisker
    • Bokeh
    • Matplotlib (not supported)
  • Violin
    • Bokeh
    • Matplotlib (not supported)
  • HexTiles
    • Bokeh
    • Matplotlib (not supported)

Not supported (i.e. no vectorized glyphs)

  • Raster
  • QuadMesh
  • Image
  • RGB
  • HSV
  • HeatMap
  • Area
  • Spread
  • Distribution
  • Bivariate
  • Text
  • HLine
  • VLine
  • Spline

Note that when I something is not supported that means that this property cannot be set as a vectorized property. Therefore to support these cases we would have to plot multiple separate artists/glyphs, which won't be handled in the first iteration of this feature.

@philippjfr

This comment has been minimized.

Copy link
Contributor Author

philippjfr commented Jun 23, 2018

This notebook also tracks what is and isn't supported yet:

https://anaconda.org/philippjfr/op_transforms/notebook

@philippjfr

This comment has been minimized.

Copy link
Contributor Author

philippjfr commented Nov 21, 2018

So I think we are now in a place where this can be merged, but there are still a few outstanding items which will need to happen in later PRs:

  • Matplotlib BarPlot needs rewriting from scratch, I'm not touching the current implementation with a 10 foot barge pole
  • SankeyPlot does not yet support style mapping, this will require some work because the styles need to be carried through the operation
  • I need to finish writing a style mapping tutorial and or integrate the material in some existing tutorial
  • Eventually we want to support applying ops directly to elements to either transform existing dimension values or add new ones
  • Write more unit tests for plot updates

Some other features this PR introduces:

  • The normalization system now handles normalizing Graph edge start/end values along the node indices which is important for consistent coloration between nodes and edges
  • The normalization system now handles computing discrete factors across plots

Another idea I've had, is that we could allow specifying a transform(s) directly on a Dimension object so when you query the dimension on an element the transform is applied. This could be very powerful as it could support a huge range of transformations:

  • Unit transforms, e.g. when querying the values would be transformed to some fixed unit
  • Coordinate system transforms
  • Storing regularly spaced log transformed values from datashader on Image coordinates (solving #2195)
  • Jitter of axis locations
  • Lots more I can't think of right now
@jbednar

This comment has been minimized.

Copy link
Contributor

jbednar commented Nov 21, 2018

Sounds great!

@philippjfr philippjfr removed the WIP label Nov 21, 2018

Philipp Rudiger added some commits Nov 22, 2018

Philipp Rudiger Philipp Rudiger
Philipp Rudiger Philipp Rudiger
Philipp Rudiger Philipp Rudiger
Philipp Rudiger Philipp Rudiger
@philippjfr

This comment has been minimized.

Copy link
Contributor Author

philippjfr commented Nov 22, 2018

Tests should pass shortly. Let's get this merged so we can get a few days of testing in before a release.

@@ -50,7 +50,7 @@
"metadata": {},
"outputs": [],
"source": [
"violin.options(height=500, width=900)"
"violin.options(height=500, width=900, violin_fill_color=('Year', str), cmap='Set1')"

This comment has been minimized.

@jlstevens

jlstevens Nov 22, 2018

Contributor

I think this is a stray instance of the tuple format...

@@ -53,7 +53,7 @@
"metadata": {},
"outputs": [],
"source": [
"boxwhisker.options(show_legend=False, width=400)"
"boxwhisker.options(show_legend=False, width=600, box_fill_color=('origin', str), cmap='Set1')"

This comment has been minimized.

@jlstevens

jlstevens Nov 22, 2018

Contributor

And another tuple...

@jlstevens

This comment has been minimized.

Copy link
Contributor

jlstevens commented Nov 22, 2018

I'm happy to merge but I think I spotted a few stray instances of the tuple format above.

@jlstevens

This comment has been minimized.

Copy link
Contributor

jlstevens commented Nov 22, 2018

Tests are now green.

I'm very happy to merge this PR as I think it will greatly simplify things throughout while exposing a more powerful way of working. It will require a fair bit of testing and of course we may well have various tweaks to make. For the time being, I'll finally go ahead and merge...nearly a year since this PR was first opened!

@jlstevens jlstevens merged commit 9fd89a6 into master Nov 22, 2018

4 checks passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details
continuous-integration/travis-ci/push The Travis CI build passed
Details
coverage/coveralls Coverage increased (+0.9%) to 89.657%
Details
s3-reference-data-cache Test data is cached.
Details

@philippjfr philippjfr deleted the ops branch Dec 13, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.