Skip to content

API: engine kw to .plot to enable selectable backends #14130

Closed

Description

@jreback

We have had some conversations in the past regarding an .plot(....., engine=) kw. in #8018
This would allow pandas to redirect the plotting to a user selectable back-end, keeping matplotlib as the default.

see chartpy here, for a way to selectively enable matplotlib, bokeh and plotly.

and generically via altair.

Missing from here is when to re-direct to seaborn.

So this issue is for discussion:

  1. should we do this
  2. implementation method and minimalist dependencies to actually do this (ideally pandas would add NO dependences itself, just import for a particular engine, raising if its not available).
  3. maybe should spin off much of the current pandas code into a separate repo (pandas-plot)?
    and this actually should be the default (rather than matplotlib), which is of course the dependency. This might allow simply removing the vast majority of the custom plotting code.

Activity

  1. added this to the 0.20.0 milestone on Aug 31, 2016
  2. jreback commented on Aug 31, 2016

    @jreback
    ContributorAuthor
  3. jorisvandenbossche commented on Sep 1, 2016

    @jorisvandenbossche
    Member

    The way that chartpy does this is by having multiple implementations of their plotting method for each engine (like pandas now has for matplotlib, but less extensive).

    So, AFAIK, to have something like this work, we would either have to implement in pandas also the other engines (which means: having more plotting related code, not something we want?), or either expect from each engine to implement some kind of plot_dataframe handling the different chart types that pandas can delegate to. And I am not sure this is something that the different engines would like to do?

  4. tacaswell commented on Sep 2, 2016

    @tacaswell
    Contributor

    With mpl we have been working to better support pandas input natively to all of our plotting routines (the data kwarg, automatic index extraction, automatic label extraction).

    It is not too hard now to write dataframe aware functions that do mostly sensible things (ex) with matplotlib. I have a suspicion that if you started from scratch and mpl 1.5+ the mpl version of the pandas plotting code would be much shorter and clearer.

    My suggestion would be to pull the current pandas plotting code out into it's own project and refactor it into functions that look like

    def some_chart_type(df, optional=backend, independent=input, *, backend=dependent, keyword=args):

    and use that as a reference implementation of the plotting API that backends need to expose to pandas for use in the plot accessor.

  5. modified the milestones: 0.20.0, on Mar 23, 2017
  6. dhirschfeld commented on Sep 26, 2017

    @dhirschfeld
    Contributor

    This may also be of interest to @santosjorge

  7. TomAugspurger commented on Sep 28, 2017

    @TomAugspurger
    Contributor

    some quick thoughts follow. Curious to hear other's.

    Pandas Plotting

    Goal: define a system for multiple backends (matplotlib, Bokeh, Plotly, Altair
    etc.) to take over DataFrame.plot.

    Note libraries can already achive this end, to an extent, with
    DataFrame.pipe(func, **kwargs). func gets the DataFrame as the first
    argument and all kwargs. It's completely up to func what happens then. This
    is about the main .plot method, which is implemented around charts.

    Overview of the implementation

    DataFrame implements .plot as a AccessorProperty. This makes .plot
    into a namespace with various plot methods. Currently, we define

    ['area', 'bar', 'barh', 'box', 'density', 'hexbin', 'hist', 'line',
     'pie', 'scatter']
    

    (scatter and hexbin are DataFrame-only; the rest are also defined on Series.plot).
    For backwards compatibility, plot is also callable, and is equivalent to .plot.line.
    These methods call matplotlib axes plotting methods.

    User API

    A user-configurable

    pandas.options.plotting.backend = {'matplotlib', 'bokeh', 'altair', 'plotly', ... }
    

    Would be the main point for users. Users would set this globally

    pd.options.plotting.backend = 'bokeh'
    

    Or use a context manager

    with pd.options_context('plotting.backend', 'bokeh'):
        df.plot(...)
    

    Backend API

    Now for the tough part.

    Changes to Pandas

    We'll refactor the current FramePlotMethods to MatplotlibFramePlotMethods.
    We'll make the actual FramePlotMethods a simple shell that

    1. looks up the currently active backend
    2. calls the appropriate method on the active backend

    So

    class FramePlotMethods:
        def line(self, x=None, y=None, **kwds):
            backend = self.get_backend()
            # _data is the DataFrame calling .plot.line
            backend.line(self._data, x=x, y=y, **kwds)

    At that point, things are entirely up to the backend. The various backends would
    implement their own FramePlotMethods (probably inherit from a base class in
    pandas that raises NotImplementedError with a nice error message saying that
    this method isn't available with this backend).

    Challenges

    API consistency

    How much should pandas care that backends accept similar keywords, behavior
    similarly, etc? I'm not sure. For the most part, we've simply adopted
    matplotlib's terminology for everything. That's probably not appropriate for
    everyone. Certain methods do have "recommended" (somewhere between required
    and optional) keyword arguments. For example .line takes an x and y. It'd
    be nice if backends could agree on those.

    Global State

    Matplotlib has the notion of a "currently active figure", and some plotting
    methods will add to that. Is there any difference between

    with pd.options_context('plotting.backend', 'bokeh'):
        df.plot()
    with pd.options_context('plotting.backend', 'matplotlib'):
        df.plot()
        
    # Any difference here?
    with pd.options_context('plotting.backend', 'bokeh'):
        df.plot()

    I don't think so (aside from the extra matplotlib plot; the bokeh plots would be
    identical). It's completely up to the backend how to handle global state between
    calls.

    Fortunately for us, pandas messed this up terribly at some point, so that
    Series.plot goes onto the currently active axes, while DataFrame.plot
    creates a new one. Users are used to diverging behavior in this area I guess :)

    registration

    I've been trying to improve pandas import time recently. Part of that involved
    removing a

    try: import matplotlib
    excpet ImportError: pass
    

    Pandas doesn't want to try / except each of the backends known to have an
    implementation. Do we require users to import bokeh.pandas, which calls a
    register_backend? That seems not great from the user's standpoint, but maybe
    necessary?

  8. TomAugspurger commented on Sep 28, 2017

    @TomAugspurger
    Contributor

    Agreed with @tacaswell here that the current implementation should be moved to the plugin system I outlined above. That would be a good test case for what other backends would need.

  9. mwaskom commented on Sep 28, 2017

    @mwaskom
    Contributor

    Missing from here is when to re-direct to seaborn

    Personally I don't think it really makes sense to consider seaborn a "backend" for pandas plotting. Seaborn seems higher in the stack than pandas, relative to the other backends. Are there particular plotting functions you had in mind for delegating to?

  10. 12 remaining items

  11. rgbkrk commented on Oct 3, 2017

    @rgbkrk
    Contributor

    To both Brians -- I may surprise you. I'd generally say that I'm a fan of format specifications. Versioned, specified in a manner that someone could write an implementation of the spec and convert it to other formats. At least in Jupyter, we need to be able to work solely with JSON because of the notebook format and messaging spec being a JSON based protocol (*). My primary care for specifications is to be able to support more than one language, which means being able to plot in non-python environments. I'd be more than happy if there were agreed upon binary format (even with arrow we can work with it on a web based frontend or serverside with node).

    (*) Caveat: kernels can send arbitrary binary blobs with messages, they're not well specced for use on the protocol though (they do get used by ipywidgets, since they intercept messages and send their own).

    Pandas luckily can return a standardized table schema thanks to @TomAugspurger and others, so I'm pretty happy in this regard for having something interoperable that isn't tied to a particular visualization. It ticks some basic boxes for the small cases that JSON formats are totally fine for.

  12. trxcllnt commented on Oct 3, 2017

    @trxcllnt

    @rgbkrk heads up, our Arrow lib is now part of the official Apache/Arrow project. The package name on npm will stay the same, intending to release 0.1.3 in the next few days 🎉

  13. rgbkrk commented on Oct 3, 2017

    @rgbkrk
    Contributor

    niiiiice

  14. TomAugspurger commented on Oct 4, 2017

    @TomAugspurger
    Contributor

    I've started a (super hacky) version of this over at master...TomAugspurger:plotting-plugin

    For engine authors, there's a base class where you can override the various .line, .scatter, etc, and a function register_engine to make pandas aware of your implementation.

  15. rs2 commented on Oct 4, 2017

    @rs2
    Contributor

    @TomAugspurger:
    Given there is DataFrame.pipe(func, **kwargs), are you suggesting that each plotting library implements callbacks to produce line, bar, scatter etc. plots for a given dataframe and a set of kwargs? This could be optimal for a number of reasons:

    • Keep majority of plotting code outside of pandas;
    • Core developers of each plotting library are probably more familiar with best practices of translating kwargs into plots;
    • pandas developers don't have to worry about changes in plotting libraries' APIs (e.g. there have been significant changes between bokeh 0.11.1 and 0.12.9).

    Challenges:

    • Get core developers' time to actually implement the callbacks.
  16. TomAugspurger commented on Oct 4, 2017

    @TomAugspurger
    Contributor

    @rs2 yeah that sounds about right (I wouldn't call them callbacks though, and it won't be using .pipe).

    Any library wishing to take over .plot can subclass the BasePlotMethods class in master...TomAugspurger:plotting-plugin and register their implementation with pandas. When the user configures pandas to use that backend, pandas will ensure that the .plot call is passed through to the correct implementation.

  17. tacaswell commented on Oct 7, 2017

    @tacaswell
    Contributor

    Could pandas also provide some helper functions for down-selecting the data frames to just the columns of interest / doing aggregations?

    I think it would also make sense for the API to provide a semantic set of inputs (ex x and y_list to plot) to guide the implementations.

    Can this be py3 only so we can use required keyword arguments?

    @ellisonbg I don't see a big difference an python api with fixed kwargs and a json schema which embeds the function name as one of the keys. If you need it is json format, it should be up to the plotting library to do that translation and export as json if required.

  18. ellisonbg commented on Oct 7, 2017

    @ellisonbg
    Contributor

    @tacaswell - great question. A few things we have observed in building things like this:

    • Imperative Python APIs tend to be leaky and end up with edge cases that are implicitly defined by behavior of a particular implementation or documentation (or lack thereof).
    • Because Python doesn't have things such as public/private, interfaces and static typing (unless you are willing to jump to python 3 only!), the contracts made by Python APIs are hard to enforce and end up relying on those informal contracts of documentation and behavior. This is very different from a JSON schema which is formal and can be validated at run time (structure and types).
    • A Python API is a stronger constraint on both Pandas and rendering libraries from the versioning perspective. A JSON schema allows both Pandas and the underlying renderers to evolve their public API to best serve users, while still following the contract of the JSON schema. A well versioned JSON schema allows a renderer to support multiple versions of that schema in a single Python package.
    • Serialization. With a JSON schema, serialization is a solved problem and users can instantly serialize their visualizations to files and send them over the network - all while allowing different renderers to be used as needed. It also makes it possible to build frontend based renderers with ease.
    • It is easy to build a set of declarative tests and examples that renderers should be able to run. With little bit of work, tests can even be autogenerated to make it easy to follow the JSON schema.

    However, if there isn't support for a JSON schema based approach, I would love to see this be python3 only so at least the python api can be strongly typed and required kw args.

  19. tacaswell commented on Oct 11, 2017

    @tacaswell
    Contributor
    • It is not clear to me that JSON schema based communication will be any better. The hard part is still the semantics, names, and intentions. We can end up with implementation specific behavior that leaks across a boxplot json schema just as easily as across a normal function call. The weirdest bits of leaky API tend to be when the meaning of a value is one parameter depends on the value of another which is something that JSON can not help us with
    • using explicit kwargs at least gets us checks that the keys are spelled correctly. It is not clear that we want these APIs to be deeply nested (that is the top level inputs to these should not be dicts). The places where I would expect the most pain would be things like "the user asked us to aggregate on a column which is not in the dataframe" which I do not think schemas can help us with (and maybe should be a basic check that pandas does for us). I am not sure I am ready to throw the duck-typing baby out with the bath water....
    • The renderer's are going to have to register with pandas, keeping around different versions and registering the correct ones based on the version of pandas installed in not hard to manage (I have managed this at my day job, it is not so bad) or to provide a version key with the registration function to pandas.
    • This is very specific to JSON based plotting libraries, what you register with pandas looks something like
    def do_boxplot(data, **kwargs):
        json = build_my_json_of_boxplot_and_validate(kwargs)
        return data, json

    For those of us with native python plotting libraries (well, fine me ;) ), this seems natural to restrict the json related things to the json libraries. It also lets you deal with any schema differences between different JSON based plotting libraries in python and give libraries a chance to do any data-preprocessing before exporting.

    • It is not clear to me how you could do better than smoke tests without a human in the loop.

    Fundamentally I think the two approaches are functionally equivalent (I'm less worried about static typing because I render in the same process in python so I get nice error messages rather than it rendering who knows where is a browser that happily eats all exceptions 😈 ). Expressing the API with a schema is reasonable (and auto-generating the pandas side of the API?), but I am not convinced that the value add is worth the effort of just writing the API to begin with.

    If this goes the JSON route mpl will just write functions that look like

    def do_boxplot(data, json_kwargs):
        json_kwargs = json.loads(json_kwargs)  # ok, I may be being pedantic here
        return realy_do_boxplot(data, **json_kwargs)

    but it seems odd to me to run an API for python libraries to talk to each other through JSON.

    I am 100% on board with this being python3 only 👍

    I should also be clear, I very much like JSON / JSON schema in general, I am just not convinced that it is the right thing to do in this case.

    @TomAugspurger Have you considered using a SimpleNamespace containing functions that look like sig(data, *, k1, k2, ...) ? Seems nicer from the implementer side to not have to subclass something from pandas and keeps us from ever seeing any of the pandas internals of how the plot accessor is implemented.

  20. modified the milestones: , 0.25.0 on Jun 21, 2019
  21. flavianh commented on Jul 27, 2019

    @flavianh

    @jreback @datapythonista I can't get it to work:

    pd.set_option('plotting.backend', 'plotly.plotly')
    df.plot(x='created_at', y='updated_at', kind='scatter')

    gives me

    ---------------------------------------------------------------------------
    PlotlyError                               Traceback (most recent call last)
    <ipython-input-32-3180a7c770d0> in <module>
          1 pd.set_option('plotting.backend', 'plotly.plotly')
    ----> 2 open_pulls_df.plot(x='created_at', y='updated_at', kind='scatter')
    
    ~/.pyenv/versions/3.7.3/lib/python3.7/site-packages/pandas/plotting/_core.py in __call__(self, *args, **kwargs)
        736         if kind in self._dataframe_kinds:
        737             if isinstance(data, ABCDataFrame):
    --> 738                 return plot_backend.plot(data, x=x, y=y, kind=kind, **kwargs)
        739             else:
        740                 raise ValueError(
    
    ~/.pyenv/versions/3.7.3/lib/python3.7/site-packages/chart_studio/plotly/plotly.py in plot(figure_or_data, validate, **plot_options)
        223     """
        224     import plotly.tools
    --> 225     figure = plotly.tools.return_figure_from_figure_or_data(figure_or_data, validate)
        226     for entry in figure['data']:
        227         if ('type' in entry) and (entry['type'] == 'scattergl'):
    
    ~/.pyenv/versions/3.7.3/lib/python3.7/site-packages/plotly/tools.py in return_figure_from_figure_or_data(figure_or_data, validate_figure)
       1130         validated = True
       1131     else:
    -> 1132         raise exceptions.PlotlyError("The `figure_or_data` positional "
       1133                                      "argument must be "
       1134                                      "`dict`-like, `list`-like, or an instance of plotly.graph_objs.Figure")
    
    PlotlyError: The `figure_or_data` positional argument must be `dict`-like, `list`-like, or an instance of plotly.graph_objs.Figure
    

    and

    pd.set_option('plotting.backend', 'plotly')
    df.plot(x='created_at', y='updated_at', kind='scatter')

    gives me

    ---------------------------------------------------------------------------
    AttributeError                            Traceback (most recent call last)
    <ipython-input-36-63d506018c44> in <module>
          1 pd.set_option('plotting.backend', 'plotly')
    ----> 2 open_pulls_df.plot(x='created_at', y='updated_at', kind='scatter')
    
    ~/.pyenv/versions/3.7.3/lib/python3.7/site-packages/pandas/plotting/_core.py in __call__(self, *args, **kwargs)
        736         if kind in self._dataframe_kinds:
        737             if isinstance(data, ABCDataFrame):
    --> 738                 return plot_backend.plot(data, x=x, y=y, kind=kind, **kwargs)
        739             else:
        740                 raise ValueError(
    
    AttributeError: module 'plotly' has no attribute 'plot'
    

    There is no doc on how it works, so I'm stuck here. Tested with pandas 0.25.0 against plotly 3.10.0 and plotly 4.0.0

  22. datapythonista commented on Jul 27, 2019

    @datapythonista
    Member

    Thanks @flavianh for reporting. Unfortunately it's not feasible to plot with arbitrary libraries, we can just plot with libraries that implement our interface. Plotly has plans to work on it, but I think the development hasn't started yet, I guess it will take few months. There is work being done in hvplot and altair to make these libraries compatible with the new API, but that is not available now.

    Afaik the only library you can use at the moment is the latest version of: https://github.com/PatrikHlobil/Pandas-Bokeh I don't think is as mature as the matplotlib plotting we provide, and I wouldn't use it in production code, but I think should be helpful for interactive plots in a notebook.

    What would be very useful is if you can open a pull request to clarify all this in the documentation you were following. So, we don't mislead other users as we did with you. Thank you in advance for it!

  23. flavianh commented on Aug 5, 2019

    @flavianh

    I think I read the changelog which mentions this very issue. I may have fast-read through the issue and thought that I could try plotly. I looked around in the documentation and it seems quite clear, especially here: "plotting.backend : str The plotting backend to use. The default value is “matplotlib”, the backend provided with pandas. Other backends can be specified by prodiving the name of the module that implements the backend. [default: matplotlib] [currently: matplotlib]".

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions