ENH: Pickle backend

[jnothman]

Feature summary

In non-interactive usages, I often want to generate numerous plots. But sometimes I will want to be able to tweak them after the fact. I would ordinarily use a script that allow me to specify an output filename (or filename format) and savefig is used to dump multiple figures; but pdf, png, svg, etc. are too lossy to tweak after the fact. If I could specify /path/to/output.pkl and the script would automatically dump a pickle, then layout could be tweaked later without having to modify the plotting script (which I may or may not have developed). Indeed, I could have generic post-processing scripts to perform layout tweaking and dump to disk again using the backend/format of my choice.

It looks like it would be quite straightforward to implement a backend to perform pickle output (although there may be questions of whether a more efficient (joblib) or more lenient (dill) pickle variant be employed).

I could implement it for myself, but I'm not sure how to then automatically register the backend, so I can use an existing script containing a savefig. (Or have I missed the existence of some matplotlibrc register_backend command?)

Do others think this would be useful?

[jklymak]

You can already pickle figure objects. Did you mean something more than this?

import numpy as np
import matplotlib.pyplot as plt
import pickle as pl

fig, ax = plt.subplots()
x = np.linspace(0,2*np.pi)
y = np.sin(x)
ax.plot(x,y)

# Save figure handle to disk
pl.dump(fig, file('figure.pickle', 'wb'))

[jnothman]

I mean that fig.savefig('out.pkl') should be supported. Thus if I have a command-line script, for instance, which saves a series of figures to disk at specified paths, I need only provide a pkl extension for it to dump something I can still interact with, or modify before rendering to image.

[anntzer]

FWIW a similar idea was rejected at #3819 (and I now agree with the rationale there).

In practice this can easily be implemented by adding a print_pkl method on the canvas (see backend_bases.py):

canvas = plt.gcf().canvas
canvas.print_pkl = lambda fname, **kwargs: pickle.dump(canvas.figure, open(fname, "wb"))
plt.savefig("...")

which you can nicely wrap in an enable_pkl or whatever function.

[jnothman]

Note that matlab similarly has saveas(figure, fmt) where fmt may be either a rendered format (png, pdf, etc) or a plot attributes dump (fig)

…

On 11 October 2017 at 13:09, Jody Klymak ***@***.***> wrote: You can already pickle figure objects. Did you mean something more than this? import numpy as npimport matplotlib.pyplot as pltimport pickle as pl fig, ax = plt.subplots() x = np.linspace(0,2*np.pi) y = np.sin(x) ax.plot(x,y) # Save figure handle to disk pl.dump(fig_handle,file('figure.pickle', 'wb')) — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#9357 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAEz6wJ5qEtwv4PJsxHUpCPiweKWNTlTks5srCNogaJpZM4P0zaB> .

[anntzer]

You can explicitly pass a format to savefig, the only question is whether to actively encourage people to save pickles. Right now we are definitely not making guarantees for cross-version compatibility of pickles, which is IMO the main argument for not encouraging their use.

[jnothman]

Yes, assuming it's reasonable to modify the generating script, then I call some enabling function. I realise I can do this for myself. And I understand the limitations of pickles. I understand if, given the general caution needed around pickles, this is not a functionality matplotlib wants to provide freely to its users. But the benefits of making scripts that generate plots in arbitrary format more reusable might be able to outweigh those limitations. The purpose here would certainly not be to use pickle as a long term storage format, but to enable it as an intermediate format for finessing and dumping to perhaps multiple other output formats. As I said in the original post, the ability to register a backend for a particular extension in matplotlibrc might be sufficient for this to be a puggable feature.

[anntzer]

Alternatively, you can just patch the print_pkl method into FigureCanvasBase (e.g. you may have a utils module that you generally import? just import matplotlib and patch it from there) and be done with it.

[jnothman]

I don't particularly like that. I suppose I can just define my own backend which extends from my default and then don't need to modify the script. It's not idea either, IMO. I'll see what I come up with. Thanks.

[tacaswell]

👎 we keep the pickle support working primarily to support multi-process work (for the case where you have and expensive to generate, but small at the end figure).

Please do not patch FigureCanvasBase on import.

See http://matplotlib.org/tutorials/introductory/usage.html#backends If you have custom sub-class of your backend of choice that provides pkl you can just do matplotlib.use("module://pickle_of_doom").

[jklymak]

I'm still not following why pickle.dump is any worse than figure.savefig.

[jnothman]

@jklymak because I already have scripts using savefig. Let's make it more tangible. I'd written my scripts to use savefig for experimental analysis. Now I want to publish some of them. Rather than go hack my code, I'd like to just specify a different path, one that ends in .pkl. Then it should behave as I seek transparently.

Thanks @tacaswell, I've realised a module:// backend is acceptable, as long as my base backend is consistent across platforms I want to apply this on. If I want to provide it as a tool for others to use, well, then, that's not really sufficient without having a way to allow the user to configure a base backend for my tool, and so the madness ensues (thus flat is better than nested).

Having a script which allows me to run other scripts in a context in which FigureCanvasBase, or at least the current backend, is patched is more portable.

But then why would you not just offer me a way to register a backend just for one extension in my matplotlibrc?

[jnothman]

PS: yes, it's useful for multiprocessing; here it is for incremental processing, where similarly the figure(s) may be expensive to generate (or depends on data that is expensive to load or aggregate).

[tacaswell]

Use http://matplotlib.org/api/backend_bases_api.html?highlight=register_backend#matplotlib.backend_bases.register_backend that should be safe to do on import of your module.

[jnothman]

Yes, that's what I hope to do. I'll close, but will ping you if the package gets off the ground and I at least find it useful.

[jnothman]

I've not yet played with it much, but here's the result of this conversation: https://github.com/jnothman/pickleback. I have a problem where a pickled figure is restored with its canvas attribute set to None and I do not understand why. Insights are welcome.