New subplot foundation for Plotly backend

[jonmmease]

Overview

This PR introduces a new foundation for combining plotly elements into a figure.

Background

Currently, the plotly backend makes use of the plotly.tools.make_subplots function to support laying out multiple elements in a single plotly figure. This function is pretty flexible when working with Cartesian trace types, but it has limited support for 3D traces and no support for other trace types. Additionally, it cannot be used recursively so it's not possible to use make_subplots to build a GridSpace figure and then make_subplots again to layout multiple GridSpace views in a single figure.

PR Notes

There is a pretty detailed commit log, but here are the two top-level goals:

Replace make_subplots with figure_grid

This PR replaces the use of make_subplots with a new figure_grid function. This function inputs a 2D list of figure dict instances, along with optional spacing arguments, and returns a new figure that is the combination of all of the input figures. It handles all plotly trace types along with annotations, shapes, and images that are specified in Cartesian axis coordinates. It also works fine recursively, so it is now possible to, for example, layout a GridSpace next to a HoloMap.

I may eventually try to roll this functionality into plotly.py directly, but it would need to be a lot more general, so I'd rather start in Holoviews where we have control over how figures are constructed.

Replace graph_objs with dicts at the Element level

This PR also changes the elements and layouts to work with dict instances rather than graph_objs instances. The graph_objs in version 3+ perform a lot more validation than in version 2, which is great when a user is directly building their own figure but it does have a performance cost. For a system like Holoviews that is performing lots of iterative construction, I think it's better to build up the figure using raw dict and list instances and then convert it to a graph_objs.Figure object for validation just before rendering.

I'm not sure if this was the best place for it, but I added the step to convert the figure dict to a Figure instance to the _figure_data method of PlotlyRenderer.

Usage highlights

Here are some examples of layouts that did not work previously

3D plots in a layout

import numpy as np
import holoviews as hv
hv.notebook_extension('plotly')

y,x = np.mgrid[-5:5, -5:5] * 0.1
heights = np.sin(x**2+y**2)
scatt3d = hv.Scatter3D(zip(x.flat,y.flat,heights.flat))
scatt3d+scatt3d.options(color_index=2, size=5, cmap='fire')

Tables in a layout

import holoviews as hv
hv.extension('plotly')

gender = ['M','M', 'M','F']
age = [10,16,13,12]
weight = [15,18,16,10]
height = [0.8,0.6,0.7,0.8]

table = hv.Table((gender, age, weight, height), ['Gender', 'Age'], ['Weight', 'Height'])
table.select(Gender='M') + table.select(Gender='M', Age=10)

Note: I still want to replace this figure_factory table with the plotly table trace, but using the figure factory implementation here shows how annotations are merged and maintained successfully.

GridSpace inside a layout

import numpy as np
import holoviews as hv
hv.extension('plotly')

def sine_curve(phase, freq):
    xvals = [0.1* i for i in range(100)]
    return hv.Curve((xvals, [np.sin(phase+freq*x) for x in xvals]))

phases      = [0, np.pi/2, np.pi, 3*np.pi/2]
frequencies = [0.5, 0.75, 1.0, 1.25]
curve_dict_2D = {(p,f):sine_curve(p,f) for p in phases for f in frequencies}

gridspace = hv.GridSpace(curve_dict_2D, kdims=['phase', 'frequency'])
hmap = hv.HoloMap(gridspace)
hmap + hv.GridSpace(hmap)

Note that the GridSpaces shared x-axes and y-axes are maintained when placed inside a larger layout.

Performance

The plotly backend had felt a bit sluggish to me compared to bokeh and matplotlib, but with these changes it's much faster that it was, and faster than the other backends in some cases. I'm not sure how to time this exactly, but here's a GIF of the display time for a layout of two 4x4 GridSpaces using each backend

Thanks for taking a look, and please let me know if you have any questions!

[jbednar]

Wow, cool!

[philippjfr]

@jonmmease You continue to amaze me! Is this in a good place to start review? We were about to declare a feature freeze, but it would be really great to get these improvements into 1.11.0.

[jonmmease]

Haha, thanks @philippjfr 🙂 Yeah, this is good to go from my end (pending CI tests passing)

[philippjfr]

We have plans to replace these since they're kind of useless (in fact we plan to migrate to pytest at some point). For now I'd recommend following a pattern where you create a baseclass like this for all plotly tests (you can look in holoviews/tests/plotting/bokeh/testplot.py to see an example):

class TestPlotlyPlot(ComparisonTestCase):

    def setUp(self):
        self.previous_backend = Store.current_backend
        self.comm_manager = bokeh_renderer.comm_manager
        bokeh_renderer.comm_manager = comms.CommManager
        if not bokeh_renderer:
            raise SkipTest("Plotly required to test plot instantiation")
        Store.current_backend = 'plotly'

    def tearDown(self):
        Store.current_backend = self.previous_backend
        bokeh_renderer.comm_manager = self.comm_manager
        Callback._callbacks = {}

[jonmmease]

It looked to me like the TestPlotlyPlotInstantiation class above follows this pattern, and this is where I put the new tests that use the plotly backend. I didn't follow the pattern for TestPlotlyFigureGrid because it's not actually testing the backend directly, just the new figure_grid util function. Does that make sense?

Would it make sense to just remove the @attr for TestPlotlyFigureGrid?

[philippjfr]

Ah, yes, totally missed that thanks. I think it's fine, I'll replace all the @attr decorators in one go anyway.

[philippjfr]

I'll point out a few things but they're all optional and I'd be happy to fix them up subsequently.

[philippjfr]

We've started converting everything to Google style docstrings. For now it's extremely inconsistent and we're still very much transitioning (e.g. I haven't even started on the plotting modules). So this is more of an FIY (which should really be recorded in some as of yet non-existent developer guide).

[philippjfr]

I'd be happy to merge this PR as is, thanks for the detailed docstrings! I'll probably have to play around a bit to get a good feel for the way the figure merge, subplot merging and figure grid code works but this seems like a very solid foundation to build on.

One thing I'd be happy to do at this point is to remove the big warning about the experimental nature of the backend. I may also work on porting the new so called dim transforms to the plotly backend to allow mapping dimension values onto any visual property and thereby replace the horrible color_index options.

[jlstevens]

First of all, thank you very much @jonmmease for your work here!

One thing I'd be happy to do at this point is to remove the big warning about the experimental nature of the backend.

Here is the current warning:

The plotly backend is experimental, and is
not supported at this time. If you would like to volunteer to help
maintain this backend by adding documentation, responding to user
issues, keeping the backend up to date as other code changes, or by
adding support for other elements, please email holoviews@gmail.com

I would like to remove this warning but I don't think we would need to debate it at all if @jonmmease were willing to volunteer as the maintainer for the plotly support. It probably wouldn't have to involve all the responsibilities listed above (e.g docs and responding to all user issues) but having someone willing to own the code would definitely help a lot.

Alternatively, we could keep the warning but edit it so it if softer (e.g still asking for help but not saying it is an experimental and unsupported feature).

[philippjfr]

We can decide on the warning separately as I'll be opening a PR with some cleanup and porting some recent features, but please do let us know if you'd be happy to have at least some ownership over the codebase. For now I'll merge.

[jbednar]

The reason for the warning is that we aren't using the Plotly backend for our own projects at this time, and so its maintenance can't be folded into anything we're working on. I'd be happy to remove the warning as long as someone were looking out for it even a little bit...

[jonmmease]

Hi @philippjfr @jlstevens @jbednar ,
Thanks for the kind words and for looking over and merging this PR so quickly! My inclination would be to hold off on removing the warning for 1.11 and then revisit it again for 1.12. I'll email you some thoughts/context on the maintainership question this evening, but my general plan at this point is to spend some time over the next month or so getting to know the code base better by picking off some items from #3196.