Implement backend options transformer

[philippjfr]

Enables plotting hvPlot output with HoloViews Matplotlib and Plotly backends.

Matplotlib
Plotly

[philippjfr]

This is now in a reasonable state but the list of transforms is not at all exhaustive. So as you take over @maximlt here's what needs to be done.

Basically I've transformed some of the most important options but there is a long tail of options which are currently not handled. As a start it would be great if you could go through all user guides and reference notebooks and visually compare the plotly and matplotlib output with the bokeh output. Another useful approach is to insert this print statement:

if opt not in el_opts.allowed_keywords:
    print(f'{opt} not supported')
    ...

and see which options are currently being skipped and either make sure they are also UNSET (if not supported) or define the appropriate transform to ensure they are supported.

[coveralls]

Coverage decreased (-0.5%) to 68.238% when pulling 483fe17 on backend_transformer into 00e1ceb on master.

[maximlt]

Here is a small write-up to clarify our intent here. As of now hvplot has a unique backend which is bokeh. hvplot's API allows to pass in kwargs to customize a bokeh plot, on the condition that they are known to holoviews. For instance to plot a dashed line one should provide the bokeh option line_dash="dashed". These options can be found by tab completion or with hvplot.help('line').

hvplot relies obviously on holoviews which itself supports two additional backends: matplotlib and plotly. It seems natural then to add support in hvplot for these alternative backends. There's however code out there that surely relies on customizing plots with bokeh kwargs. Where this happens, hvplot should try its best to transform the bokeh options provided to options understood by the targeted backend. For instance, the line_dash equivalent option for Matplotlib would be linestyle="dashed". Under the hood a mapping is used to store the options that can be linked or not between bokeh and matplotlib/plotly (TBD), and when they can be linked a transform function.

This allows for instance a user to turn a notebook with bokeh plots to a notebook with matplotlib plots, while partly preserving the plot options originally defined for the bokeh backend. Note that this is not a dead-end, the underlying plot object can be accessed with hv.render and further customized.

# These two lines change the backend to matplotlib
import holoviews as hv
hv.extension('matplotlib')

import hvplot.pandas

df = ...
plot = df.hvplot.line(x='x', y='y', line_dash='dashed')
# Display plot in a cell
plot

# Get the Matplotlib figure to further customize it if required
fig = hv.render(plot)
...
fig.savefig('fig.png')

What has not yet been done in this work is to directly deal with matplotlib or plotly options, e.g. what to do with df.hvplot.line(linestyle="dashed"). This could be supported in principle. As of now this would result in a warning being emitted due to an option unknown to hvplot being used.

[maximlt]

Here's a list of the current issues observed while running all the notebooks with the PR in its current state.

General:

• date ticks are usually too long and not readable since partially overlapping => holoviews: use pandas smart date formatting in matplotlib backend => Opened an issue on Holoviews (Nicely render time-based axis for the Matplotlib backend holoviews#5112)
• Can't get the line_width option to work => hvplot or holoviews? See the comment below (Implement backend options transformer #653 (comment)) => Fixed
• - [ ] Cannot use a column to drive the transparency of the points of a scatter plot with alpha='column_name' => Opened a Holoviews issue (Allow mapping of a dimension to the alpha style option with Matplotlib and Plotly holoviews#5223)

User guides

• Plotting:
  • Bar plot with two y variables -> Display 'Year, Variable' in the x label ('Year' being x here), 'Variable' shouldn't be displayed there Opened an issue on Holoviews (The x-label of a stacked Bars plot should not display the last/stacked level (Matplotlib only)  holoviews#5113)
  • hexbin: doesn't display an image and just returns :HexTiles [airtime,arrdelay], note this uses a dask df => working with holoviews so hvplot issue. See the comment below (Implement backend options transformer #653 (comment))
  • heatmap: display the annotations by default (default in holoviews) => Opened an issue on holoviews (Suggesting that by default a Heatmap has no annotations with the Matplotlib backend holoviews#5108)
  • hist: error with by=... but that could also be because of dask again => Opened an issue on holoviews (TypeError when rendering a Holomap with Matplotlib whose data is a Dask DataFrame holoviews#5111)
  • box: figure with no plot when by=..., dask again => Opened an issue on holoviews (Empty Box-Whisker plot with the Matplotlib backend when vdims has NaNs holoviews#5120)
  • datashader: lots of unecessary space around the plot => Opened an issue on holoviews (Default aspect/padding/size of datashaded plots with the Matplotlib backend is wrong holoviews#5121)
  • violin: displaying the axes only, but again could be because of dask => Opened an issue on holoviews (Empty Violin plot with the Matplotlib backend holoviews#5124)
• Pandas API:
  • overlay of a bar plot and a hline raises a TypeError => Opened an issue in holoviews (Bars plot with a label overlaid with a HLine raises a TypeError with the Matplotlib backend holoviews#5125)
  • scatter: no colormap visible when c='c' because 'c' means cyan => Opened an issue in hvplot (The c parameter of a scatter plot should respect the field name even if it's a color name #675)
  • df.plot.scatter(x='a', y='b', color=hv.dim('c'), s=50, colorbar=True) => Opened an issue in hvplot (Cannot use hv.dim() to declare the c color parameter of a scatter plot #674)
  • scatter_matrix, andrew_curves, parallel_coordinates: not working, options are not transformed in the PR
  • error bars: not working well at all, the error lines are next to the bars => Opened an issue in holoviews (Overlay of Bars and ErrorBars renders the error bars next to the bars with the Matplotlib backend holoviews#5126)
  • colormap: colormap='Greens' not working, colormap not respected => Opened an issue on holoviews (Equivalent of the cmap style option for a Matplotlib Bars plot with a colormap holoviews#5193)
• streams: not working df.groupby('y').sum().hvplot.bar(x='y') leads to a ZeroDivisionError => Opened an issue on holoviews (Streaming groupby plot raises an error with the Matplotib backend holoviews#5202)
• subplots: grid size really small and cannot be increased => Wrong, it can be increased after the fact with `.opts(fig_inches=...) (and/or fig_size)
• time series: violin matplotlib plots have a cycled colormap while bokeh plots are just blue => Opened an issue in Holoviews (No cycle colormap for the Violin plot with Matplotlib holoviews#5114)
• Geographic:
  • Error when plotting points and tiles => hvplot, see the comment below (Implement backend options transformer #653 (comment))
  • combination of datashade=True and project=True returns an upside-down map => Opened an issue in geoviews (datashade a Polygons element renders it upside down with the Matplotlib backend geoviews#541)

Reference

• heatmap: annotations displayed by default => Opened an issue in holoviews (Suggesting that by default a Heatmap has no annotations with the Matplotlib backend holoviews#5108)
• ohlc: not working at all
• violin: cmap not respected
• contour and contourf: not working at all (clim=(None, None), is that expected?) => Opened a PR directly in hvplot (Update the default internal value of clim to support Matplotlib #679)
• image: cartopy hangs when downloading the coastline shapefile (this is being fixed by cartopy) => Fixed by cartopy
• quadmesh: the layout is kind of weird, too much padding around the plot => Should be fixed by Ensure HoloViews matplotlib output uses tight layout panel#2920
• geopandas points: Issue with tiles: 'No plotting class for Tiles type and matplotlib backend found' => see the comment below (Implement backend options transformer #653 (comment))

Extras

• legend_position not supported for a scatter plot => Opened a HoloViews issue (Support legend_position for Scatter plots with Matplotlib and Plotly holoviews#5221)

[maximlt]

EDIT 22/03/2022: Fixed

@philippjfr about the issue with line_width not being taken into account, I think I found why but I'm not sure what's the appropriate fix.

At this line

hvplot/hvplot/converter.py

Line 958 in 483fe17

el_options = element.opts.get(backend='bokeh').kwargs

the options I get for a Curve plot when I have declared that I want a Matplotlib plot (import holoviews as hv; hv.extension('matplotlib'); import hvplot.pandas) are:

OrderedDict([('axiswise', False),
             ('color', Cycle()),
             ('framewise', True),
             ('height', 300),
             ('labelled', ['x', 'y']),
             ('line_width', 20),
             ('linewidth', 2),
             ('logx', False),
             ('logy', False),
             ('muted_alpha', 0.2),
             ('responsive', False),
             ('shared_axes', True),
             ('show_grid', False),
             ('show_legend', True),
             ('tools', ['hover']),
             ('width', 700)])

They include linewidth=2, which is the default option for a Curve for the Matplotlib backend in holoviews:
https://github.com/holoviz/holoviews/blob/5e058cadb7c9817216e6eeafb2588ee5c5a5bc8c/holoviews/plotting/mpl/__init__.py#L231

Is it expected that .get() returns options from both backends?

[maximlt]

In the Plotting guide the call to create a Hexbin plot (flights.hvplot.hexbin(x='airtime', y='arrdelay', width=600, height=500, logz=True)) doesn't display any image, it just returns :HexTiles [airtime,arrdelay].

Since the following code correctly displays an image with the Matplotlib backend I assume this is a hvplot issue.

hv.HexTiles(flights, kdims=['airtime', 'arrdelay']).opts(fig_size=200, logz=True, colorbar=True)

The object returned by hvplot has this .dataset property:

:Dataset   [index]   (year,month,day,dayofweek,dep_time,crs_dep_time,arr_time,crs_arr_time,carrier,flight_num,tail_num,actual_elapsed_time,crs_elapsed_time,airtime,arrdelay,depdelay,origin,dest,distance,taxi_in,taxi_out,cancelled,cancellation_code,diverted,carrier_delay,weather_delay,nas_delay,security_delay,late_aircraft_delay)

while the one returned by holoviews has:

:Dataset   [airtime,arrdelay]   (year,month,day,dayofweek,dep_time,crs_dep_time,arr_time,crs_arr_time,carrier,flight_num,tail_num,actual_elapsed_time,crs_elapsed_time,depdelay,origin,dest,distance,taxi_in,taxi_out,cancelled,cancellation_code,diverted,carrier_delay,weather_delay,nas_delay,security_delay,late_aircraft_delay)

[maximlt]

Executing this:

fig = airports.hvplot.points('Longitude', 'Latitude', geo=True, color='red', alpha=0.2,
                       xlim=(-180, -30), ylim=(0, 72), tiles='ESRI')

works correctly with the Bokeh backend but fails with a horrible error message (holoviz/geoviews#540) with the Matplotlib one.

Interestingly xlim and ylim appear to have been projected to Web Mercartor, while the crs of the Points element is still PlateCarree.

:Overlay
 | Options(axiswise=False, framewise=False)
   .Tiles.I  :Tiles   [x,y]
    | Options(axiswise=False, framewise=False)
   .Points.I :Points   [Longitude,Latitude]
    | Options(alpha=0.2, axiswise=False, cmap='kbc_r', color='red', data_aspect=1, framewise=True,
    |         logx=False, logy=False, marker='o', s=5.477225575051661, show_frame=True, show_grid=False,
    |         show_legend=True, xlim=(-19981848.597392607, -3339584.8351176973), ylim=(0.1113194902619854,
    |         11753184.255101029))

Anyway, Tiles elements don't have a matplotlib implementation in holoviews, so no background tiles would be plotted.

One way to adapt the code to support plotting background tiles with the Matplotlib backend may be to use the WMTS elements provided by geoviews instead of the Tiles elements provided by holoviews, given that the latter have no projection.

[maximlt]

Plotly issues:

General

• No support of the data_aspect option => Opened a Holoviews issue (Add data_aspect support for the Plotly backend holoviews#5225)
• Bivariate plots doesn't accept bandwidth, cut and levels => Opened a Holoviews issue (Align the plot parameters of Bivariate with Plotly to the ones supported by the other backends holoviews#5224)
• The legend is (always?) shown while it's not with Bokeh or Matplotlib => Opened a Holoviews issue (show_legend doesn't have the same effect given the selected backend holoviews#5217)
• legend_position not supported for a scatter plot => Opened a HoloViews issue (Support legend_position for Scatter plots with Matplotlib and Plotly holoviews#5221)
• Colorizing the points of a Scatter plot with c='column_name' raises an error => Opened a Holoviews issue (Colorizing the points of a Scatter plot with the Plotly backend raises an error holoviews#5222)
• Cannot use a column to drive the transparency of the points of a scatter plot with alpha='column_name' => Opened a Holoviews issue (Allow mapping of a dimension to the alpha style option with Matplotlib and Plotly holoviews#5223)
• Area plots don't support alpha -> Opened a Holoviews issue (Add alpha styling support to Area and Bar plots with plotly holoviews#5226)

Plotting notebook:

• hexbin plot (HexTiles in holoviews) not yet available with the plotly backend => Opened a Holoviews issue (HexTiles plot with Plotly holoviews#5219)
• KeyError when generating a bivariate plot => Opened a Holoviews issue (KeyError when creating a bivariate plot from a pandas dataframe with Plotly holoviews#5220)

[maximlt]

@jlstevens could you please review the last changes wrt compatibility?

[jlstevens]

Other than the suggestion to update the parameter docstring, the latest changes look fine to me!

[jlstevens]

Don't think this is really related to this PR, but the lack of an x-axis label bothers me. What are the units? Absolute time or presumably time deltas?

[maximlt]

This is the exact same example as pandas' one, which doesn't mean it's perfect!
https://pandas.pydata.org/docs/reference/api/pandas.plotting.lag_plot.html

[jlstevens]

I would add a sentence along the lines of 'This notebook demonstrates the use of the Matplotlib plotting backend, the equivalent notebook demonstrating the Plotly backend may be found HERE (link)'

[jlstevens]

Same as for matplotlib. One sentence should be enough to link the contents of the notebook to the notebook title.

[jlstevens]

Made some suggestions for the docs but overall this looks good!

[maximlt]

@jlstevens thanks for the review! I applied your suggestions in the last two commits.

[jlstevens]

Tests look like they are passing and I'm happy to see it merge from my side!