You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
API: engine kw to .plot to enable selectable backends #14130
We have had some conversations in the past regarding an .plot(....., engine=) kw. in #8018
This would allow pandas to redirect the plotting to a user selectable back-end, keeping matplotlib as the default.
see chartpy here, for a way to selectively enable matplotlib, bokeh and plotly.
Missing from here is when to re-direct to seaborn.
So this issue is for discussion:
should we do this
implementation method and minimalist dependencies to actually do this (ideally pandas would add NO dependences itself, just import for a particular engine, raising if its not available).
maybe should spin off much of the current pandas code into a separate repo (pandas-plot)?
and this actually should be the default (rather than matplotlib), which is of course the dependency. This might allow simply removing the vast majority of the custom plotting code.
The way that chartpy does this is by having multiple implementations of their plotting method for each engine (like pandas now has for matplotlib, but less extensive).
So, AFAIK, to have something like this work, we would either have to implement in pandas also the other engines (which means: having more plotting related code, not something we want?), or either expect from each engine to implement some kind of plot_dataframe handling the different chart types that pandas can delegate to. And I am not sure this is something that the different engines would like to do?
With mpl we have been working to better support pandas input natively to all of our plotting routines (the data kwarg, automatic index extraction, automatic label extraction).
It is not too hard now to write dataframe aware functions that do mostly sensible things (ex) with matplotlib. I have a suspicion that if you started from scratch and mpl 1.5+ the mpl version of the pandas plotting code would be much shorter and clearer.
My suggestion would be to pull the current pandas plotting code out into it's own project and refactor it into functions that look like
some quick thoughts follow. Curious to hear other's.
Pandas Plotting
Goal: define a system for multiple backends (matplotlib, Bokeh, Plotly, Altair
etc.) to take over DataFrame.plot.
Note libraries can already achive this end, to an extent, with DataFrame.pipe(func, **kwargs). func gets the DataFrame as the first
argument and all kwargs. It's completely up to func what happens then. This
is about the main .plot method, which is implemented around charts.
Overview of the implementation
DataFrame implements .plot as a AccessorProperty. This makes .plot
into a namespace with various plot methods. Currently, we define
(scatter and hexbin are DataFrame-only; the rest are also defined on Series.plot).
For backwards compatibility, plot is also callable, and is equivalent to .plot.line.
These methods call matplotlib axes plotting methods.
Would be the main point for users. Users would set this globally
pd.options.plotting.backend = 'bokeh'
Or use a context manager
with pd.options_context('plotting.backend', 'bokeh'):
df.plot(...)
Backend API
Now for the tough part.
Changes to Pandas
We'll refactor the current FramePlotMethods to MatplotlibFramePlotMethods.
We'll make the actual FramePlotMethods a simple shell that
looks up the currently active backend
calls the appropriate method on the active backend
So
classFramePlotMethods:
defline(self, x=None, y=None, **kwds):
backend=self.get_backend()
# _data is the DataFrame calling .plot.linebackend.line(self._data, x=x, y=y, **kwds)
At that point, things are entirely up to the backend. The various backends would
implement their own FramePlotMethods (probably inherit from a base class in
pandas that raises NotImplementedError with a nice error message saying that
this method isn't available with this backend).
Challenges
API consistency
How much should pandas care that backends accept similar keywords, behavior
similarly, etc? I'm not sure. For the most part, we've simply adopted
matplotlib's terminology for everything. That's probably not appropriate for
everyone. Certain methods do have "recommended" (somewhere between required
and optional) keyword arguments. For example .line takes an x and y. It'd
be nice if backends could agree on those.
Global State
Matplotlib has the notion of a "currently active figure", and some plotting
methods will add to that. Is there any difference between
withpd.options_context('plotting.backend', 'matplotlib'):
df.plot()
# Any difference here?withpd.options_context('plotting.backend', 'bokeh'):
df.plot()
I don't think so (aside from the extra matplotlib plot; the bokeh plots would be
identical). It's completely up to the backend how to handle global state between
calls.
Fortunately for us, pandas messed this up terribly at some point, so that Series.plot goes onto the currently active axes, while DataFrame.plot
creates a new one. Users are used to diverging behavior in this area I guess :)
registration
I've been trying to improve pandas import time recently. Part of that involved
removing a
try: import matplotlib
excpet ImportError: pass
Pandas doesn't want to try / except each of the backends known to have an
implementation. Do we require users to import bokeh.pandas, which calls a register_backend? That seems not great from the user's standpoint, but maybe
necessary?
Agreed with @tacaswellhere that the current implementation should be moved to the plugin system I outlined above. That would be a good test case for what other backends would need.
Personally I don't think it really makes sense to consider seaborn a "backend" for pandas plotting. Seaborn seems higher in the stack than pandas, relative to the other backends. Are there particular plotting functions you had in mind for delegating to?
To both Brians -- I may surprise you. I'd generally say that I'm a fan of format specifications. Versioned, specified in a manner that someone could write an implementation of the spec and convert it to other formats. At least in Jupyter, we need to be able to work solely with JSON because of the notebook format and messaging spec being a JSON based protocol (*). My primary care for specifications is to be able to support more than one language, which means being able to plot in non-python environments. I'd be more than happy if there were agreed upon binary format (even with arrow we can work with it on a web based frontend or serverside with node).
(*) Caveat: kernels can send arbitrary binary blobs with messages, they're not well specced for use on the protocol though (they do get used by ipywidgets, since they intercept messages and send their own).
Pandas luckily can return a standardized table schema thanks to @TomAugspurger and others, so I'm pretty happy in this regard for having something interoperable that isn't tied to a particular visualization. It ticks some basic boxes for the small cases that JSON formats are totally fine for.
@rgbkrk heads up, our Arrow lib is now part of the official Apache/Arrow project. The package name on npm will stay the same, intending to release 0.1.3 in the next few days 🎉
For engine authors, there's a base class where you can override the various .line, .scatter, etc, and a function register_engine to make pandas aware of your implementation.
@TomAugspurger:
Given there is DataFrame.pipe(func, **kwargs), are you suggesting that each plotting library implements callbacks to produce line, bar, scatter etc. plots for a given dataframe and a set of kwargs? This could be optimal for a number of reasons:
Keep majority of plotting code outside of pandas;
Core developers of each plotting library are probably more familiar with best practices of translating kwargs into plots;
pandas developers don't have to worry about changes in plotting libraries' APIs (e.g. there have been significant changes between bokeh 0.11.1 and 0.12.9).
Challenges:
Get core developers' time to actually implement the callbacks.
@rs2 yeah that sounds about right (I wouldn't call them callbacks though, and it won't be using .pipe).
Any library wishing to take over .plot can subclass the BasePlotMethods class in master...TomAugspurger:plotting-plugin and register their implementation with pandas. When the user configures pandas to use that backend, pandas will ensure that the .plot call is passed through to the correct implementation.
Could pandas also provide some helper functions for down-selecting the data frames to just the columns of interest / doing aggregations?
I think it would also make sense for the API to provide a semantic set of inputs (ex x and y_list to plot) to guide the implementations.
Can this be py3 only so we can use required keyword arguments?
@ellisonbg I don't see a big difference an python api with fixed kwargs and a json schema which embeds the function name as one of the keys. If you need it is json format, it should be up to the plotting library to do that translation and export as json if required.
@tacaswell - great question. A few things we have observed in building things like this:
Imperative Python APIs tend to be leaky and end up with edge cases that are implicitly defined by behavior of a particular implementation or documentation (or lack thereof).
Because Python doesn't have things such as public/private, interfaces and static typing (unless you are willing to jump to python 3 only!), the contracts made by Python APIs are hard to enforce and end up relying on those informal contracts of documentation and behavior. This is very different from a JSON schema which is formal and can be validated at run time (structure and types).
A Python API is a stronger constraint on both Pandas and rendering libraries from the versioning perspective. A JSON schema allows both Pandas and the underlying renderers to evolve their public API to best serve users, while still following the contract of the JSON schema. A well versioned JSON schema allows a renderer to support multiple versions of that schema in a single Python package.
Serialization. With a JSON schema, serialization is a solved problem and users can instantly serialize their visualizations to files and send them over the network - all while allowing different renderers to be used as needed. It also makes it possible to build frontend based renderers with ease.
It is easy to build a set of declarative tests and examples that renderers should be able to run. With little bit of work, tests can even be autogenerated to make it easy to follow the JSON schema.
However, if there isn't support for a JSON schema based approach, I would love to see this be python3 only so at least the python api can be strongly typed and required kw args.
It is not clear to me that JSON schema based communication will be any better. The hard part is still the semantics, names, and intentions. We can end up with implementation specific behavior that leaks across a boxplot json schema just as easily as across a normal function call. The weirdest bits of leaky API tend to be when the meaning of a value is one parameter depends on the value of another which is something that JSON can not help us with
using explicit kwargs at least gets us checks that the keys are spelled correctly. It is not clear that we want these APIs to be deeply nested (that is the top level inputs to these should not be dicts). The places where I would expect the most pain would be things like "the user asked us to aggregate on a column which is not in the dataframe" which I do not think schemas can help us with (and maybe should be a basic check that pandas does for us). I am not sure I am ready to throw the duck-typing baby out with the bath water....
The renderer's are going to have to register with pandas, keeping around different versions and registering the correct ones based on the version of pandas installed in not hard to manage (I have managed this at my day job, it is not so bad) or to provide a version key with the registration function to pandas.
This is very specific to JSON based plotting libraries, what you register with pandas looks something like
For those of us with native python plotting libraries (well, fine me ;) ), this seems natural to restrict the json related things to the json libraries. It also lets you deal with any schema differences between different JSON based plotting libraries in python and give libraries a chance to do any data-preprocessing before exporting.
It is not clear to me how you could do better than smoke tests without a human in the loop.
Fundamentally I think the two approaches are functionally equivalent (I'm less worried about static typing because I render in the same process in python so I get nice error messages rather than it rendering who knows where is a browser that happily eats all exceptions 😈 ). Expressing the API with a schema is reasonable (and auto-generating the pandas side of the API?), but I am not convinced that the value add is worth the effort of just writing the API to begin with.
If this goes the JSON route mpl will just write functions that look like
defdo_boxplot(data, json_kwargs):
json_kwargs=json.loads(json_kwargs) # ok, I may be being pedantic herereturnrealy_do_boxplot(data, **json_kwargs)
but it seems odd to me to run an API for python libraries to talk to each other through JSON.
I am 100% on board with this being python3 only 👍
I should also be clear, I very much like JSON / JSON schema in general, I am just not convinced that it is the right thing to do in this case.
@TomAugspurger Have you considered using a SimpleNamespace containing functions that look like sig(data, *, k1, k2, ...) ? Seems nicer from the implementer side to not have to subclass something from pandas and keeps us from ever seeing any of the pandas internals of how the plot accessor is implemented.
Thanks @flavianh for reporting. Unfortunately it's not feasible to plot with arbitrary libraries, we can just plot with libraries that implement our interface. Plotly has plans to work on it, but I think the development hasn't started yet, I guess it will take few months. There is work being done in hvplot and altair to make these libraries compatible with the new API, but that is not available now.
Afaik the only library you can use at the moment is the latest version of: https://github.com/PatrikHlobil/Pandas-Bokeh I don't think is as mature as the matplotlib plotting we provide, and I wouldn't use it in production code, but I think should be helpful for interactive plots in a notebook.
What would be very useful is if you can open a pull request to clarify all this in the documentation you were following. So, we don't mislead other users as we did with you. Thank you in advance for it!
I think I read the changelog which mentions this very issue. I may have fast-read through the issue and thought that I could try plotly. I looked around in the documentation and it seems quite clear, especially here: "plotting.backend : str The plotting backend to use. The default value is “matplotlib”, the backend provided with pandas. Other backends can be specified by prodiving the name of the module that implements the backend. [default: matplotlib] [currently: matplotlib]".
Activity
jreback commentedon Aug 31, 2016
@jorisvandenbossche
@sinhrks
@wesm
@TomAugspurger
@shoyer
cc @ellisonbg
cc @tacaswell
cc @mdboom
cc @mwaskom
cc @pzwang
cc @bryevdv
jorisvandenbossche commentedon Sep 1, 2016
The way that
chartpydoes this is by having multiple implementations of their plotting method for each engine (like pandas now has for matplotlib, but less extensive).So, AFAIK, to have something like this work, we would either have to implement in pandas also the other engines (which means: having more plotting related code, not something we want?), or either expect from each engine to implement some kind of
plot_dataframehandling the different chart types that pandas can delegate to. And I am not sure this is something that the different engines would like to do?tacaswell commentedon Sep 2, 2016
With mpl we have been working to better support pandas input natively to all of our plotting routines (the
datakwarg, automatic index extraction, automatic label extraction).It is not too hard now to write dataframe aware functions that do mostly sensible things (ex) with matplotlib. I have a suspicion that if you started from scratch and mpl 1.5+ the mpl version of the pandas plotting code would be much shorter and clearer.
My suggestion would be to pull the current pandas plotting code out into it's own project and refactor it into functions that look like
and use that as a reference implementation of the plotting API that backends need to expose to pandas for use in the plot accessor.
dhirschfeld commentedon Sep 26, 2017
This may also be of interest to @santosjorge
TomAugspurger commentedon Sep 28, 2017
some quick thoughts follow. Curious to hear other's.
Pandas Plotting
Goal: define a system for multiple backends (matplotlib, Bokeh, Plotly, Altair
etc.) to take over
DataFrame.plot.Note libraries can already achive this end, to an extent, with
DataFrame.pipe(func, **kwargs).funcgets theDataFrameas the firstargument and all kwargs. It's completely up to
funcwhat happens then. Thisis about the main
.plotmethod, which is implemented around charts.Overview of the implementation
DataFrameimplements.plotas aAccessorProperty. This makes.plotinto a namespace with various plot methods. Currently, we define
(scatter and hexbin are DataFrame-only; the rest are also defined on
Series.plot).For backwards compatibility,
plotis also callable, and is equivalent to.plot.line.These methods call
matplotlibaxes plotting methods.User API
A user-configurable
Would be the main point for users. Users would set this globally
Or use a context manager
Backend API
Now for the tough part.
Changes to Pandas
We'll refactor the current
FramePlotMethodstoMatplotlibFramePlotMethods.We'll make the actual
FramePlotMethodsa simple shell thatSo
At that point, things are entirely up to the backend. The various backends would
implement their own
FramePlotMethods(probably inherit from a base class inpandas that raises
NotImplementedErrorwith a nice error message saying thatthis method isn't available with this backend).
Challenges
API consistency
How much should pandas care that backends accept similar keywords, behavior
similarly, etc? I'm not sure. For the most part, we've simply adopted
matplotlib's terminology for everything. That's probably not appropriate for
everyone. Certain methods do have "recommended" (somewhere between required
and optional) keyword arguments. For example
.linetakes anxandy. It'dbe nice if backends could agree on those.
Global State
Matplotlib has the notion of a "currently active figure", and some plotting
methods will add to that. Is there any difference between
I don't think so (aside from the extra matplotlib plot; the bokeh plots would be
identical). It's completely up to the backend how to handle global state between
calls.
Fortunately for us, pandas messed this up terribly at some point, so that
Series.plotgoes onto the currently active axes, whileDataFrame.plotcreates a new one. Users are used to diverging behavior in this area I guess :)
registration
I've been trying to improve pandas import time recently. Part of that involved
removing a
Pandas doesn't want to try / except each of the backends known to have an
implementation. Do we require users to
import bokeh.pandas, which calls aregister_backend? That seems not great from the user's standpoint, but maybenecessary?
TomAugspurger commentedon Sep 28, 2017
Agreed with @tacaswell here that the current implementation should be moved to the plugin system I outlined above. That would be a good test case for what other backends would need.
mwaskom commentedon Sep 28, 2017
Personally I don't think it really makes sense to consider seaborn a "backend" for pandas plotting. Seaborn seems higher in the stack than pandas, relative to the other backends. Are there particular plotting functions you had in mind for delegating to?
12 remaining items
rgbkrk commentedon Oct 3, 2017
To both Brians -- I may surprise you. I'd generally say that I'm a fan of format specifications. Versioned, specified in a manner that someone could write an implementation of the spec and convert it to other formats. At least in Jupyter, we need to be able to work solely with JSON because of the notebook format and messaging spec being a JSON based protocol (*). My primary care for specifications is to be able to support more than one language, which means being able to plot in non-python environments. I'd be more than happy if there were agreed upon binary format (even with arrow we can work with it on a web based frontend or serverside with node).
(*) Caveat: kernels can send arbitrary binary blobs with messages, they're not well specced for use on the protocol though (they do get used by ipywidgets, since they intercept messages and send their own).
Pandas luckily can return a standardized table schema thanks to @TomAugspurger and others, so I'm pretty happy in this regard for having something interoperable that isn't tied to a particular visualization. It ticks some basic boxes for the small cases that JSON formats are totally fine for.
trxcllnt commentedon Oct 3, 2017
@rgbkrk heads up, our Arrow lib is now part of the official Apache/Arrow project. The package name on npm will stay the same, intending to release 0.1.3 in the next few days 🎉
rgbkrk commentedon Oct 3, 2017
niiiiice
TomAugspurger commentedon Oct 4, 2017
I've started a (super hacky) version of this over at master...TomAugspurger:plotting-plugin
For engine authors, there's a base class where you can override the various
.line,.scatter, etc, and a functionregister_engineto make pandas aware of your implementation.rs2 commentedon Oct 4, 2017
@TomAugspurger:
Given there is
DataFrame.pipe(func, **kwargs), are you suggesting that each plotting library implements callbacks to produce line, bar, scatter etc. plots for a given dataframe and a set ofkwargs? This could be optimal for a number of reasons:pandas;pandasdevelopers don't have to worry about changes in plotting libraries' APIs (e.g. there have been significant changes betweenbokeh0.11.1 and 0.12.9).Challenges:
TomAugspurger commentedon Oct 4, 2017
@rs2 yeah that sounds about right (I wouldn't call them callbacks though, and it won't be using
.pipe).Any library wishing to take over
.plotcan subclass theBasePlotMethodsclass in master...TomAugspurger:plotting-plugin and register their implementation with pandas. When the user configures pandas to use that backend, pandas will ensure that the.plotcall is passed through to the correct implementation.tacaswell commentedon Oct 7, 2017
Could pandas also provide some helper functions for down-selecting the data frames to just the columns of interest / doing aggregations?
I think it would also make sense for the API to provide a semantic set of inputs (ex
xandy_listtoplot) to guide the implementations.Can this be py3 only so we can use required keyword arguments?
@ellisonbg I don't see a big difference an python api with fixed kwargs and a json schema which embeds the function name as one of the keys. If you need it is json format, it should be up to the plotting library to do that translation and export as json if required.
ellisonbg commentedon Oct 7, 2017
@tacaswell - great question. A few things we have observed in building things like this:
However, if there isn't support for a JSON schema based approach, I would love to see this be python3 only so at least the python api can be strongly typed and required kw args.
tacaswell commentedon Oct 11, 2017
boxplotjson schema just as easily as across a normal function call. The weirdest bits of leaky API tend to be when the meaning of a value is one parameter depends on the value of another which is something that JSON can not help us withFor those of us with native python plotting libraries (well, fine me ;) ), this seems natural to restrict the json related things to the json libraries. It also lets you deal with any schema differences between different JSON based plotting libraries in python and give libraries a chance to do any data-preprocessing before exporting.
Fundamentally I think the two approaches are functionally equivalent (I'm less worried about static typing because I render in the same process in python so I get nice error messages rather than it rendering who knows where is a browser that happily eats all exceptions 😈 ). Expressing the API with a schema is reasonable (and auto-generating the pandas side of the API?), but I am not convinced that the value add is worth the effort of just writing the API to begin with.
If this goes the JSON route mpl will just write functions that look like
but it seems odd to me to run an API for python libraries to talk to each other through JSON.
I am 100% on board with this being python3 only 👍
I should also be clear, I very much like JSON / JSON schema in general, I am just not convinced that it is the right thing to do in this case.
@TomAugspurger Have you considered using a
SimpleNamespacecontaining functions that look likesig(data, *, k1, k2, ...)? Seems nicer from the implementer side to not have to subclass something from pandas and keeps us from ever seeing any of the pandas internals of how the plot accessor is implemented.flavianh commentedon Jul 27, 2019
@jreback @datapythonista I can't get it to work:
gives me
and
gives me
There is no doc on how it works, so I'm stuck here. Tested with pandas 0.25.0 against plotly 3.10.0 and plotly 4.0.0
datapythonista commentedon Jul 27, 2019
Thanks @flavianh for reporting. Unfortunately it's not feasible to plot with arbitrary libraries, we can just plot with libraries that implement our interface. Plotly has plans to work on it, but I think the development hasn't started yet, I guess it will take few months. There is work being done in hvplot and altair to make these libraries compatible with the new API, but that is not available now.
Afaik the only library you can use at the moment is the latest version of: https://github.com/PatrikHlobil/Pandas-Bokeh I don't think is as mature as the matplotlib plotting we provide, and I wouldn't use it in production code, but I think should be helpful for interactive plots in a notebook.
What would be very useful is if you can open a pull request to clarify all this in the documentation you were following. So, we don't mislead other users as we did with you. Thank you in advance for it!
flavianh commentedon Aug 5, 2019
I think I read the changelog which mentions this very issue. I may have fast-read through the issue and thought that I could try plotly. I looked around in the documentation and it seems quite clear, especially here: "plotting.backend : str The plotting backend to use. The default value is “matplotlib”, the backend provided with pandas. Other backends can be specified by prodiving the name of the module that implements the backend. [default: matplotlib] [currently: matplotlib]".