Adding png image return for inline backend figures with _repr_html_ #16788

tdpetrou · 2020-03-16T15:56:10Z

PR Summary

This will enable figures to be displayed in a jupyter notebook without ever having to run %matplotlib inline or import matplotlib.pyplot.

The initial issue was created here #16782.

I'm sure the code will need to be modified quite a bit to work with more cases, but I wanted to get the ball started on this important issue for me.

I don't know how to test this. Are there special tests for jupyter notebook output?

PR Checklist

Has Pytest style unit tests
Code is Flake 8 compliant
New features are documented, with examples if plot related
Documentation is sphinx and numpydoc compliant
Added an entry to doc/users/next_whats_new/ if major new feature (follow instructions in README.rst there)
Documented in doc/api/api_changes.rst if API changed in a backward-incompatible way

anntzer · 2020-03-16T15:59:33Z

lib/matplotlib/figure.py

@@ -375,6 +375,15 @@ def _repr_html_(self):
            from matplotlib.backends import backend_webagg
            return backend_webagg.ipython_inline_display(self)

+        if mpl.rcParams['backend'] == 'module://ipykernel.pylab.backend_inline':


this should (likely?) check the canvas type (as for webagg), because rcParams["backend"] may change?

That makes sense - should we have a figure output for all backends? And just choose the image format based on that backend?

Instead of adding more coditional paths here we should make this defer to a _repr_html_ on the canvas.

@tacaswell Would adding a single _repr_html_ to backend_bases.FigureCanvaseBase be enough?

No, because for the interactive backends we do not want to replace them with a dead png.

But this would only get triggered if a figure was the output of the last line of a cell in a notebook, which would (likely) only be done intentionally.

story645 · 2020-03-16T16:14:04Z

Awesome, thanks for opening this issue!

tacaswell · 2020-03-16T17:08:09Z

lib/matplotlib/figure.py

@@ -375,6 +375,15 @@ def _repr_html_(self):
            from matplotlib.backends import backend_webagg
            return backend_webagg.ipython_inline_display(self)

+        if mpl.rcParams['backend'] == 'module://ipykernel.pylab.backend_inline':


Instead of adding more coditional paths here we should make this defer to a _repr_html_ on the canvas.

tdpetrou · 2020-03-16T23:27:16Z

I changed the approach completely and added a _repr_html_ to backend_bases.FigureCanvasBase. It gets the default format type and returns that format as HTML. It defaults to png whenever the format is not one of png, svg, jpg, or pdf. I used the <embed> tag for pdfs.

In Figure, the _repr_html_ defers to the canvas _repr_html_ for all cases. I moved the existing code to the backend_webagg.FigureCanvasWebAgg class under its own _repr_html_.

anntzer · 2020-03-16T23:35:56Z

lib/matplotlib/backend_bases.py

+
+        fmt = self.get_default_filetype()
+        dpi = self.figure.dpi
+        if fmt == 'retina':


is there really a backend that defines such a default filetype?

FigureCanvasBase defaults to rcParams['savefig.format']. svg, pdf, pgf, ps all have this method implemented.

Yes, but the format won't be 'retina'.

Thank you for this. I see what I need to do. Get the ipython shell and check for retina as the figure_format. I would also need to get other settings like bbox_inches from there.

Regarding retina - it implicitly doubles the DPI, but this gives a false impression of what a saved image will look like. I think it would be better to change the rcparams. Even better would be to have a backend titled 'inline_real' and set the DPI to the DPI of the monitor (and adjust the sizes of all fonts, lines, tick lines, etc.. accordingly.).

I am somewhat familiar with the problems of retina (well, on a theoretical PoV). My point is just that get_default_filetype() never returns (AFAIK) "retina"; that info is simply stashed elsewhere.

anntzer · 2020-03-17T19:04:26Z

lib/matplotlib/backend_bases.py

+
+        mimetype_dict = {'png': 'image/png', 'svg': 'image/svg+xml', 
+                         'jpg': 'image/jpeg', 'pdf': 'application/pdf'}
+        if fmt not in mimetype_dict:


falling back to png here seems wrong/too late (it could well not be a png file, indeed).

As of now, running %matplotlib inline uses default figure_format as png. So regardless of the backend, a png will be produced.

anntzer · 2020-03-17T19:04:55Z

lib/matplotlib/backend_bases.py

+        }
+
+        bytes_io = io.BytesIO()
+        self.print_figure(bytes_io, **kw)


you can just pass the kwargs here (having a separate dict doesn't seem to buy much)

and you need to force the format here. again, bytes_io could easily contain a postscript image here, for example.

anntzer · 2020-03-17T19:05:24Z

lib/matplotlib/backend_bases.py

+        self.print_figure(bytes_io, **kw)
+        raw_bytes = bytes_io.getvalue()
+
+        mimetype_dict = {'png': 'image/png', 'svg': 'image/svg+xml', 


see #14218 for how to get the mapping from the stdlib

tdpetrou · 2020-03-17T20:31:43Z

Made more changes to FigureCanvasBase._repr_html_ to account for retina display.

Currently, when retina is chosen as the figure format with %config InlineBackend.figure_format = 'retina' a png will always be output. This functionality is changed in this pull request to use the get_default_filetype of the canvas.

The bbox_inches are set to 'tight' whenever %matplotlib inline has not been run. They are set to the InlineBackend.print_figure_kwargs['bbox_inches'] when inline activated.

All backends usercParams['savefig.format'] whenever canvas.get_default_filetype is not implemented.

Backend nbagg is still interactive (if %matplotlib notebook was run) as it appears that ipython's display formatter gets triggered first which starts the application (I think?).

anntzer · 2020-03-17T21:16:31Z

Ah, I missed the fact that ipython had an additional "retina" format. Ugh.

ImportanceOfBeingErnest · 2020-03-17T22:07:48Z

lib/matplotlib/backend_bases.py

@@ -1595,6 +1595,73 @@ def __init__(self, figure):
        self.toolbar = None  # NavigationToolbar2 will set me
        self._is_idle_drawing = False

+    def _repr_html_(self):
+        if not self.figure.axes and not self.figure.lines:
+            return


What if the figure only contains a Rectangle an arrow and a Text?

True, I copied that from elsewhere. Is there a clean way to determine if the figure has changed at all?

Check https://matplotlib.org/faq/howto_faq.html#check-whether-a-figure-is-empty
However, maybe even completely empty figures should return some html? After all, you get what you ask for, right?

What do you mean "changed"?

I agree with @ImportanceOfBeingErnest if we are going to have a repr, it should always show something even if that something is an empty image.

ImportanceOfBeingErnest · 2020-03-17T22:09:46Z

lib/matplotlib/backend_bases.py

+        is_retina = False
+
+        import IPython
+        ip = IPython.get_ipython()


This would fail whenever IPython is not available.

Wouldn't _repr_html_ only get triggered for ipython?

I guess IPython invented this; but in principle anything can try to call such method if it's available.

The other way you can check this is if 'IPython' in sys.modules. That way if the user has not previously imported IPython we won't do it for them.

ImportanceOfBeingErnest · 2020-03-17T22:57:54Z

This seems to pretty much reimplement the Ipykernel inline backend into matplotlib. I'm not convinced that this is the right approach, because it uses the IPython configuration system to determine the right parameters and thereby introduces a dependency on that config system which I don't think should be there.

What about adding a Figure.to_html(**savefigkw) method instead?

tdpetrou · 2020-03-17T23:28:00Z

@ImportanceOfBeingErnest The point of this is to work with jupyter notebooks whenever a figure is the last line of a cell without the need for %matplotlib inline.

The only reason IPython is accessed is to check whether the display is retina and what bbox_inches is set to.

If you had a to_html method then you'd have to hook it into ipython somehow, which would use similar code.

ImportanceOfBeingErnest · 2020-03-17T23:49:03Z

I think my problem is really about the two-way dependency introduced here. If IPython/IPykernel decides to introduce a new figure format "SchweineGulasch", is matplotlib expected to take it over as well? If they decide to change the default bbox_inches parameter, should matplotlib adapt to it?

But if we don't, then people will complain that the figures they produce with the IPykernel inline backend look different from the ones they create via matplotlib's _repr_html_.

tdpetrou · 2020-03-18T00:17:53Z

Good point. It would be nice if ipython was made completely independent from matplotlib and dropped all of its code that had to do with it. matplotlib could then handle everything with the importing of pyplot and _repr_html_. Would be a much cleaner approach.

outdated

tacaswell · 2020-03-18T02:44:27Z

lib/matplotlib/backend_bases.py

+                dpi = dpi * 2
+                fmt = self.get_default_filetype()
+        else:
+            bbox_inches = 'tight' # how to let user choose self.figure.bbox_inches?


This just should not be passed or set to None. That IPython inline defaults to 'tight' has caused a tremendous amount of confusion over the years.

tacaswell · 2020-03-18T02:50:01Z

I again agree with @ImportanceOfBeingErnest , the default implementation on the canvas base-class should not special case the inline backend and IPyhon's configuration system (just embedding a png at the native size would be fine). The support for that could be done in a companion PR into IPython to move their logic to the correct place. It seems fair for people who want the fancier controls from IPython have to use the IPython sourced backend.

There will also need to be a companion PR into ipympl so that we correctly defer to their canvas as well.

I am very 👍 on this PR in principle, but the technical interaction between Matplotlib and IPython/jupyter has always been complicated and we need to make sure that we are making it simpler for the devs while making it more streamlined for the users.

tdpetrou · 2020-03-18T14:15:24Z

Thanks @tacaswell for the feedback. I've changed approaches again and now defer all html output to ipython if InlineBackend is in the configurables list. This greatly simplifies things. (Is there a better way to check for this?)

Otherwise, the default canvas format gets returned as HTML.

Essentially, this method will only be useful for those who don't run %matplotlib and don't import pyplot. As soon as either of those happen, ipython is in full control.

Carreau · 2020-03-18T15:28:41Z

lib/matplotlib/backend_bases.py

+            ib_list = [c for c in ip.configurables 
+                       if 'InlineBackend' in type(c).__name__]
+            if ib_list:
+                return


Just dived into IPython itself. It looks like we are also storing the sate on a function attribute (!!!).

In [1]: from IPython.core.pylabtools import configure_inline_support In [2]: configure_inline_support.current_backend --------------------------------------------------------------------------- AttributeError Traceback (most recent call last) <ipython-input-2-eec19e858e54> in <module> ----> 1 configure_inline_support.current_backend AttributeError: 'function' object has no attribute 'current_backend' In [3]: %matplotlib qt In [4]: configure_inline_support.current_backend Out[4]: 'other' In [5]: %matplotlib inline In [6]: configure_inline_support.current_backend Out[6]: 'inline'

Not sure it is better though.

Carreau · 2020-03-18T15:34:11Z

Definitively the separation IPython/Matplotlib is not great, and I'm happy to try to get better behavior on the IPython codebase.

The concern I have with __repr_html__ is that you may end up with duplicated figures if the last line of a cell return a figure. We went to relatively great length to make sure figurer get displayed when they are not the last in the cell; so if you end up having a call at end-of-cell, that happen to also return a fig/ax, then you will risk double display.

I don't like having to special case matplotlib either, and admittedly there is no really good way to know wether we are in inline mode or not. At some points, we likely want to sit-down for a couple of hours list the current behavior and plan for a progressive cleanup over the next few years.

tdpetrou · 2020-03-18T16:08:54Z

Definitively the separation IPython/Matplotlib is not great, and I'm happy to try to get better behavior on the IPython codebase.

Heh, I'm about to submit a post on the matplotlib discourse stating how great this would be :)

Regarding the double display: Currently, I experience the double display whenever I have a plt.command in the cell along with fig at the bottom of the same cell. This new _repr_html_ would trigger double display if pyplot was imported or %matplotlib run and the last line was a plt.command that returned a figure (or another library that returned a figure using pyplot). I think this would be very rare.

Agreed, the current behavior is pretty convoluted and I think can be greatly simplified.

jklymak · 2020-07-16T16:45:27Z

Closing in lieu of #17891, but feel free to reopen if I've misunderstood...

added png return for _repr_html_ for inline figures

1e156c8

anntzer reviewed Mar 16, 2020

View reviewed changes

story645 linked an issue Mar 16, 2020 that may be closed by this pull request

Add _repr_html_ for inline backend to eliminate need for %matplotlib inline #16782

Closed

tacaswell previously requested changes Mar 16, 2020

View reviewed changes

created _repr_html_ for FigureCanvasBase

2a82c32

using calculated dpi

71d4c19

anntzer reviewed Mar 16, 2020

View reviewed changes

anntzer reviewed Mar 17, 2020

View reviewed changes

corrected for retina

efd3168

ImportanceOfBeingErnest reviewed Mar 17, 2020

View reviewed changes

tacaswell reviewed Mar 18, 2020

View reviewed changes

tacaswell added this to the v3.3.0 milestone Mar 18, 2020

defer to ipython for html whenever possible

b09de9c

Carreau reviewed Mar 18, 2020

View reviewed changes

QuLogic modified the milestones: v3.3.0, v3.4.0 May 5, 2020

bdice mentioned this pull request Jul 11, 2020

Add _repr_png_ for figures with no configured IPython backend #17891

Closed

6 tasks

jklymak closed this Jul 16, 2020

QuLogic added the status: duplicate label Mar 16, 2021

QuLogic modified the milestones: v3.4.0, v3.5.0 Mar 16, 2021

Adding png image return for inline backend figures with _repr_html_ #16788

Adding png image return for inline backend figures with _repr_html_ #16788

Conversation

tdpetrou commented Mar 16, 2020 • edited Loading

PR Summary

PR Checklist

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

story645 commented Mar 16, 2020

Choose a reason for hiding this comment

tdpetrou commented Mar 16, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tdpetrou commented Mar 17, 2020 • edited Loading

anntzer commented Mar 17, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ImportanceOfBeingErnest commented Mar 17, 2020

tdpetrou commented Mar 17, 2020

ImportanceOfBeingErnest commented Mar 17, 2020

tdpetrou commented Mar 18, 2020

Choose a reason for hiding this comment

tacaswell commented Mar 18, 2020

tdpetrou commented Mar 18, 2020

Choose a reason for hiding this comment

Carreau commented Mar 18, 2020

tdpetrou commented Mar 18, 2020

jklymak commented Jul 16, 2020

tdpetrou commented Mar 16, 2020 •

edited

Loading

tdpetrou commented Mar 17, 2020 •

edited

Loading