Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

New subplot foundation for Plotly backend #3255

Merged
merged 11 commits into from Dec 4, 2018

Conversation

Projects
None yet
4 participants
@jonmmease
Copy link
Collaborator

jonmmease commented Dec 4, 2018

Overview

This PR introduces a new foundation for combining plotly elements into a figure.

Background

Currently, the plotly backend makes use of the plotly.tools.make_subplots function to support laying out multiple elements in a single plotly figure. This function is pretty flexible when working with Cartesian trace types, but it has limited support for 3D traces and no support for other trace types. Additionally, it cannot be used recursively so it's not possible to use make_subplots to build a GridSpace figure and then make_subplots again to layout multiple GridSpace views in a single figure.

PR Notes

There is a pretty detailed commit log, but here are the two top-level goals:

Replace make_subplots with figure_grid

This PR replaces the use of make_subplots with a new figure_grid function. This function inputs a 2D list of figure dict instances, along with optional spacing arguments, and returns a new figure that is the combination of all of the input figures. It handles all plotly trace types along with annotations, shapes, and images that are specified in Cartesian axis coordinates. It also works fine recursively, so it is now possible to, for example, layout a GridSpace next to a HoloMap.

I may eventually try to roll this functionality into plotly.py directly, but it would need to be a lot more general, so I'd rather start in Holoviews where we have control over how figures are constructed.

Replace graph_objs with dicts at the Element level

This PR also changes the elements and layouts to work with dict instances rather than graph_objs instances. The graph_objs in version 3+ perform a lot more validation than in version 2, which is great when a user is directly building their own figure but it does have a performance cost. For a system like Holoviews that is performing lots of iterative construction, I think it's better to build up the figure using raw dict and list instances and then convert it to a graph_objs.Figure object for validation just before rendering.

I'm not sure if this was the best place for it, but I added the step to convert the figure dict to a Figure instance to the _figure_data method of PlotlyRenderer.

Usage highlights

Here are some examples of layouts that did not work previously

3D plots in a layout

import numpy as np
import holoviews as hv
hv.notebook_extension('plotly')

y,x = np.mgrid[-5:5, -5:5] * 0.1
heights = np.sin(x**2+y**2)
scatt3d = hv.Scatter3D(zip(x.flat,y.flat,heights.flat))
scatt3d+scatt3d.options(color_index=2, size=5, cmap='fire')

newplot 11

Tables in a layout

import holoviews as hv
hv.extension('plotly')

gender = ['M','M', 'M','F']
age = [10,16,13,12]
weight = [15,18,16,10]
height = [0.8,0.6,0.7,0.8]

table = hv.Table((gender, age, weight, height), ['Gender', 'Age'], ['Weight', 'Height'])
table.select(Gender='M') + table.select(Gender='M', Age=10)

newplot 12

Note: I still want to replace this figure_factory table with the plotly table trace, but using the figure factory implementation here shows how annotations are merged and maintained successfully.

GridSpace inside a layout

import numpy as np
import holoviews as hv
hv.extension('plotly')

def sine_curve(phase, freq):
    xvals = [0.1* i for i in range(100)]
    return hv.Curve((xvals, [np.sin(phase+freq*x) for x in xvals]))

phases      = [0, np.pi/2, np.pi, 3*np.pi/2]
frequencies = [0.5, 0.75, 1.0, 1.25]
curve_dict_2D = {(p,f):sine_curve(p,f) for p in phases for f in frequencies}

gridspace = hv.GridSpace(curve_dict_2D, kdims=['phase', 'frequency'])
hmap = hv.HoloMap(gridspace)
hmap + hv.GridSpace(hmap)

gridspace_hmap

Note that the GridSpaces shared x-axes and y-axes are maintained when placed inside a larger layout.

Performance

The plotly backend had felt a bit sluggish to me compared to bokeh and matplotlib, but with these changes it's much faster that it was, and faster than the other backends in some cases. I'm not sure how to time this exactly, but here's a GIF of the display time for a layout of two 4x4 GridSpaces using each backend

gridspace_performance

Thanks for taking a look, and please let me know if you have any questions!

jonmmease added some commits Dec 3, 2018

Added new figure_grid utility function
This builds a figure out of a 2D grid of sub figures. It overcomes the limitations
of the make_subplots method because it handles all trace types and it works
recursively
Convert plotly elements from using `graph_objs` to plain dict instances
This gives us more freedom in the low-level figure manipulation utilities
(those added in `holoviews/plotting/plotly/util.py`) and it is somewhat faster
as well
Convert figure dict to `plotly.graph_objs.Figure` object during rende…
…ring

This way we still get full property validation, but it only happens once at the very
end.
Convert figure dict to `graph_objs.Figure` in testing
This will trigger property validation errors during testing if any of the
backend operations produce invalid plotly figure dicts
@jbednar

This comment has been minimized.

Copy link
Contributor

jbednar commented Dec 4, 2018

Wow, cool!

@philippjfr philippjfr added the feature label Dec 4, 2018

@philippjfr

This comment has been minimized.

Copy link
Contributor

philippjfr commented Dec 4, 2018

@jonmmease You continue to amaze me! Is this in a good place to start review? We were about to declare a feature freeze, but it would be really great to get these improvements into 1.11.0.

@jonmmease

This comment has been minimized.

Copy link
Collaborator Author

jonmmease commented Dec 4, 2018

Haha, thanks @philippjfr 🙂 Yeah, this is good to go from my end (pending CI tests passing)

@@ -123,3 +193,126 @@ def test_layout_instantiate_subplots_transposed(self):
plot = plotly_renderer.get_plot(layout(plot=dict(transpose=True)))
positions = [(0, 0), (0, 1), (1, 0), (1, 1), (2, 0), (2, 1), (3, 0), (3, 1)]
self.assertEqual(sorted(plot.subplots.keys()), positions)


@attr(optional=1)

This comment has been minimized.

@philippjfr

philippjfr Dec 4, 2018

Contributor

We have plans to replace these since they're kind of useless (in fact we plan to migrate to pytest at some point). For now I'd recommend following a pattern where you create a baseclass like this for all plotly tests (you can look in holoviews/tests/plotting/bokeh/testplot.py to see an example):

class TestPlotlyPlot(ComparisonTestCase):

    def setUp(self):
        self.previous_backend = Store.current_backend
        self.comm_manager = bokeh_renderer.comm_manager
        bokeh_renderer.comm_manager = comms.CommManager
        if not bokeh_renderer:
            raise SkipTest("Plotly required to test plot instantiation")
        Store.current_backend = 'plotly'

    def tearDown(self):
        Store.current_backend = self.previous_backend
        bokeh_renderer.comm_manager = self.comm_manager
        Callback._callbacks = {}

This comment has been minimized.

@jonmmease

jonmmease Dec 4, 2018

Author Collaborator

It looked to me like the TestPlotlyPlotInstantiation class above follows this pattern, and this is where I put the new tests that use the plotly backend. I didn't follow the pattern for TestPlotlyFigureGrid because it's not actually testing the backend directly, just the new figure_grid util function. Does that make sense?

Would it make sense to just remove the @attr for TestPlotlyFigureGrid?

This comment has been minimized.

@philippjfr

philippjfr Dec 4, 2018

Contributor

Ah, yes, totally missed that thanks. I think it's fine, I'll replace all the @attr decorators in one go anyway.

@philippjfr

This comment has been minimized.

Copy link
Contributor

philippjfr commented Dec 4, 2018

I'll point out a few things but they're all optional and I'd be happy to fix them up subsequently.


def _get_subplot_number(subplot_val):
"""
Extract the subplot number from a subplot value string.

This comment has been minimized.

@philippjfr

philippjfr Dec 4, 2018

Contributor

We've started converting everything to Google style docstrings. For now it's extremely inconsistent and we're still very much transitioning (e.g. I haven't even started on the plotting modules). So this is more of an FIY (which should really be recorded in some as of yet non-existent developer guide).

@philippjfr

This comment has been minimized.

Copy link
Contributor

philippjfr commented Dec 4, 2018

I'd be happy to merge this PR as is, thanks for the detailed docstrings! I'll probably have to play around a bit to get a good feel for the way the figure merge, subplot merging and figure grid code works but this seems like a very solid foundation to build on.

One thing I'd be happy to do at this point is to remove the big warning about the experimental nature of the backend. I may also work on porting the new so called dim transforms to the plotly backend to allow mapping dimension values onto any visual property and thereby replace the horrible color_index options.

@jlstevens

This comment has been minimized.

Copy link
Contributor

jlstevens commented Dec 4, 2018

First of all, thank you very much @jonmmease for your work here!

One thing I'd be happy to do at this point is to remove the big warning about the experimental nature of the backend.

Here is the current warning:

The plotly backend is experimental, and is
not supported at this time. If you would like to volunteer to help
maintain this backend by adding documentation, responding to user
issues, keeping the backend up to date as other code changes, or by
adding support for other elements, please email holoviews@gmail.com

I would like to remove this warning but I don't think we would need to debate it at all if @jonmmease were willing to volunteer as the maintainer for the plotly support. It probably wouldn't have to involve all the responsibilities listed above (e.g docs and responding to all user issues) but having someone willing to own the code would definitely help a lot.

Alternatively, we could keep the warning but edit it so it if softer (e.g still asking for help but not saying it is an experimental and unsupported feature).

@philippjfr

This comment has been minimized.

Copy link
Contributor

philippjfr commented Dec 4, 2018

We can decide on the warning separately as I'll be opening a PR with some cleanup and porting some recent features, but please do let us know if you'd be happy to have at least some ownership over the codebase. For now I'll merge.

@philippjfr philippjfr merged commit dea2f0b into pyviz:master Dec 4, 2018

2 checks passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details
s3-reference-data-cache Test data is cached.
Details
@jbednar

This comment has been minimized.

Copy link
Contributor

jbednar commented Dec 4, 2018

The reason for the warning is that we aren't using the Plotly backend for our own projects at this time, and so its maintenance can't be folded into anything we're working on. I'd be happy to remove the warning as long as someone were looking out for it even a little bit...

@jonmmease

This comment has been minimized.

Copy link
Collaborator Author

jonmmease commented Dec 4, 2018

Hi @philippjfr @jlstevens @jbednar ,
Thanks for the kind words and for looking over and merging this PR so quickly! My inclination would be to hold off on removing the warning for 1.11 and then revisit it again for 1.12. I'll email you some thoughts/context on the maintainership question this evening, but my general plan at this point is to spend some time over the next month or so getting to know the code base better by picking off some items from #3196.

@jonmmease jonmmease referenced this pull request Dec 9, 2018

Open

Master list of potential Plotly backend enhancements #3196

6 of 21 tasks complete
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.