Improve interaction with Matplotlib

[tacaswell]

This PR is mostly a feature request/suggestion but it was easier to describe it via code that in text. So here is a draft PR (rather than an issue with an inline diff).

---

This is a sketch of how to improve the interaction with Matplotlib.

The first idea is to directly create Figure objects rather than relying on pyplot. This de-couples the UI from the global state that is the pyplot figure registry and ensures that you never get surprise GUI windows popping up (or a memory leak) when pyplot selects a GUI backend.

The second suggestion is to have the object return by the context manager be able to create multiple figures (not actually implemented).

The third is to support saving the figures as png instead of svg (not actually implemented).

[falkoschindler]

Hi @tacaswell, thanks for the suggestion!
Can you give an example on how we would use the new interface compared to the current API?
And what exactly is the problem with importing pyplot in the user code?

[tacaswell]

https://github.com/matplotlib/mpl-gui?tab=readme-ov-file#motivation is a slightly longer explanation, but the core of it is that is a bunch of global state you do not need and runs the risk of starting up a GUI event loop that (may) allocate memory that will only be release when the event loop is run (but you never run the event loop!).

The usage would look something like:

with MplFigure(fmt='png') as fig_factory:
   fig = fig_factory.figure()
   ax = fig.subplots()
   # ax.....
   fig2 = fig_factory.figure()
   ax2 = fig2.subpots()
   # ax2....

[falkoschindler]

Thanks, @tacaswell!

Let me comment on the individual ideas:

1. Create Figure objects. You can do that already with the existing API:

   with ui.pyplot(figsize=(3, 2)) as plot:
       fig = plot.fig
       ax = fig.gca()
       x = np.linspace(0.0, 5.0)
       y = np.cos(2 * np.pi * x) * np.exp(-x)
       ax.plot(x, y, '-')

2. Create multiple figures. I don't immediately see the benefit. But we could discuss it in a separate feature request.
3. Saving figures in PNG format. This might be a useful feature, but requires generating, storing and serving a separate file. Due to the added complexity I'm not to eager to do that. But we can discuss it in a separate feature request.

[tacaswell]

The point of not using pyplot is that nicegui should not use pyplot (because it provides no real benefit and a whole lot of costs).

If you only ever want to do one figure, then might as well just return it from the context manager (I guess if you want multiple figures in the UI you use multiple context managers?).

for 3, you can base64 encode it and set it as a data url (e.g. what is done in jupyternotebooks) https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/Data_URLs .

[falkoschindler]

The point of not using pyplot is that nicegui should not use pyplot (because it provides no real benefit and a whole lot of costs).

I think I didn't get this point yet. What is the benefit of using matplotlib.figure instead of matplotlib.pyplot?

for 3, you can base64 encode it and set it as a data url (e.g. what is done in jupyternotebooks) https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/Data_URLs .

Sure, that's possible.

[tacaswell]

1. if you end up on a system where the automatic backend picks a GUI backend you may end up with objects that can to be fully garbage collected until you run the GUI event loop (even though you have a bunch of logic to make sure that you close the figure with pyplot)
2. Fundamentally, pyplot is a bunch of helper functions to make working in an interactive prompt easier and are not a good fit for writing libraries on top of. If you do not use pyplot you can drop the _auto_close(...) logic because there would be no global state so the Matplotlib objects would be cleaned up in normal order as would any other Python object.
3. if the users pass num=... which gets passed through to plt.figure then they user will be able to get the same instance in multiple places. Maybe this is desired, but I suspect it is more likely it will cause problems if multiple handlers are running at once (not 100% sure that is possible as I have not traced all the asnyc details).

Put the other way, it is not clear to me what benefit you are getting from pyplot.

[falkoschindler]

Oh, I see. I never really thought about the relation between matplotlib and plotly. Just for reference, here's ChatGPT's explanation on this matter:

The relationship between Matplotlib and pyplot can be understood as follows:

• Matplotlib is the whole package; it's a plotting library that contains various modules including pyplot.
• pyplot is one of the modules in Matplotlib (specifically, matplotlib.pyplot), which provides a set of functions that make Matplotlib work like MATLAB, offering a stateful, procedural interface. This means that pyplot keeps track of things like the current figure and plotting area, and the plotting functions are directed to the current axes (please note that "axes" here refers to part of the figure where data is plotted, and not the plural of "axis").

Now I understand, why it is weird to use a ui.pyplot element if all you want is a matplotlib figure (and not the MATLAB-like interface). For those (like me) who are used to pyplot, ui.pyplot might still be a handy element. So a new element like your MplFigures seems like a good solution. I tend to call it ui.matplotlib and suggest an API like this:

with ui.matplotlib(figsize=(3, 2)) as fig:
    ax = fig.gca()
    x = np.linspace(0.0, 5.0)
    y = np.cos(2 * np.pi * x) * np.exp(-x)
    ax.plot(x, y, '-')

Or would ui.mpl_figure be a better name?

I guess if you want multiple figures in the UI you use multiple context managers?

Yes, I think so. Directly providing a figure instead of a figure factory simplifies the API quite a bit.

[falkoschindler]

I just updated the code to reflect the new API

with ui.matplotlib(figsize=(3, 2)) as fig:
    ...

There's one issue though: A UI element is supposed to return itself when entering its context. But here we return an MPL figure, which causes a type error. Maybe we need to change it to

with ui.matplotlib(figsize=(3, 2)) as mpl:
    fig = mpl.figure
    ...

or

with ui.matplotlib() as mpl:
    fig = mpl.figure(figsize=(3, 2))
    ...

or simply

fig = mpl.Figure(figsize=(3, 2))
ui.matplotlib(fig)

The latter is more in line with ui.plotly. Since we don't need to build the figure inside some context like with pyplot, we can create it independently and pass it to the UI element.

[falkoschindler]

@rodja I finished the new ui.matplotlib element. In contrast to our earlier idea we need the figure context to know when up update the rendered HTML.

Now this is the new usage compared to ui.pyplot:

import numpy as np
from matplotlib import pyplot as plt

from nicegui import ui

x = np.linspace(0.0, 5.0)
y = np.cos(2 * np.pi * x) * np.exp(-x)

with ui.pyplot(figsize=(3, 2)):
    plt.plot(x, y, '-')

with ui.matplotlib(figsize=(3, 2)).figure as fig:
    fig.gca().plot(x, y, '-')

Note how ui.matplotlib doesn't need any imports other than ui.

[tacaswell]

Why not

with ui.mpl_figure(figsize=(3, 2)) as fig:
    fig.gca().plot(x, y, '-')

?

[falkoschindler]

@tacaswell Here I explained the problem: #2553 (comment)

[tacaswell]

Ah, sorry for the noise (and my failure to re-read the full thread). That makes sense.