Support blitting in webagg backend #19059

ianhi · 2020-12-03T03:57:59Z

PR Summary

Closes: #4288
Closes: matplotlib/ipympl#228
Supersedes: #9240

I replaced the second Agg renderer by storing the previous buffer in a private attribute. Removing the second renderer as suggested by @tacaswell eliminates the the flickering issues noted in #9240 (comment)

I also tried out a modified version of the test case by @mbewley in #9240 (comment) and found that performance was improved considerably when blitting was enabled.

from imp import reload

import matplotlib
reload(matplotlib)

matplotlib.use('nbagg')

import matplotlib.backends.backend_nbagg
reload(matplotlib.backends.backend_nbagg)
import matplotlib.backends.backend_webagg_core
reload(matplotlib.backends.backend_webagg_core)


import numpy as np
import matplotlib.pyplot as plt
import matplotlib.patches as mpatches
IMSIZE = 7000
RED = (1,0,0,1)
GREEN = (0,1,0,1)

plt.ion()
class ExampleToggle:
    
    def __init__(self):
        self.fig = plt.figure()
        self.ax = self.fig.add_subplot(111)
        self.canvas = self.ax.figure.canvas

        self.im = np.random.randint(0, 255, [IMSIZE,IMSIZE, 3]).astype('uint8')
        self.tmp_colour = RED
        self.rect = mpatches.Rectangle([0, 0], 200, 200, edgecolor='yellow', facecolor=self.tmp_colour, linewidth=0.5, animated=True)
        self.ax.add_patch(self.rect)
        
        self.canvas.mpl_connect('button_press_event', self.clicked)
        self.ax.imshow(self.im)
        self.canvas.draw()
        self._been_clicked = False

    def clicked(self, event):
        if not self._been_clicked:
            self.background = self.canvas.copy_from_bbox(self.ax.bbox)    
            self._been_clicked = True

        if self.tmp_colour == RED:
            self.tmp_colour = GREEN
        elif self.tmp_colour == GREEN:
            self.tmp_colour = RED
        self.rect.set_facecolor(self.tmp_colour)
        self.canvas.restore_region(self.background)
        self.ax.draw_artist(self.rect)
        self.canvas.blit(self.ax.bbox)
et = ExampleToggle()

(the modification was to use the self._been_clicked as I was finding that otherwise the figure was resizing and the bbox was incorrect)

Unlike the previous PR this does not need to implement copy_from_bbox as that is inherited from backend_agg.FigureCanvasAgg.

I didn't add any new tests but I did try the UAT notebook and found that the flickering mentioned is is resolved by this approach. Should we consider adding the above blit test for perforamance to the UAT notebook?

One issue is that at times the animation can get tripped up and fail to properly blit. Though I believe that this is an issue that was revealed by, rather than caused by, this PR. My theory is that maybe new messages reach the frontend before it finishes a redraw and then the frontend trips over itself? This is something I've noticed on occasion with ipympl as well, even without blitting. The result looks like this, though if I increase the interval of the animation then this pathology goes away:

with interval=32

with interval=100

PR Checklist

[UAT] Has pytest style unit tests (and pytest passes).
Is Flake 8 compliant (run flake8 on changed files to check).
[N/A (i think?)] New features are documented, with examples if plot related.
[N/A] Documentation is sphinx and numpydoc compliant (the docs should build without error).
Conforms to Matplotlib style conventions (install flake8-docstrings and run flake8 --docstring-convention=all).
New features have an entry in doc/users/next_whats_new/ (follow instructions in README.rst there).
API changes documented in doc/api/next_api_changes/ (follow instructions in README.rst there).

would this get an API note or a new feature? or neither?

Replaced the second Agg renderer by storing the previous buffer in a private attribute. Removing the second renderer eliminates the the flickering issues noted in: matplotlib#9240 (comment) Also, did not need to implement copy_from_bbox as that is inherited from backend_agg.FigureCanvasAgg

ianhi · 2020-12-03T04:10:36Z

I didn't realize there was such a nice UAT added in the previous PR. I've extracted that and added it here:

import itertools

cnt = itertools.count()
bg = None

def onclick_handle(event):
    """Should draw elevating green line on each mouse click"""
    global bg
    if bg is None:
        bg = ax.figure.canvas.copy_from_bbox(ax.bbox)        
    ax.figure.canvas.restore_region(bg)

    cur_y = (next(cnt) % 10) * 0.1
    ln.set_ydata([cur_y, cur_y])
    ax.draw_artist(ln)
    ax.figure.canvas.blit(ax.bbox)


fig, ax = plt.subplots()
ax.plot([0, 1], [0, 1], 'r')
ln, = ax.plot([0, 1], [0, 0], 'g', animated=True)
plt.show()
ax.figure.canvas.draw()

ax.figure.canvas.mpl_connect('button_press_event', onclick_handle)

Here as well if I click to fast I can get smearing:

tacaswell · 2020-12-03T06:57:00Z

Thanks for working on this @ianhi :)

@danielballan has also noticed this form of glitching in ipympl and is interested in fixing it. I agree the issue is that via some mechanism one or more of the diff frames are being lost or processed out of order. We either need to make the front ends smart enough to detect this (by tacking a frame number on to everything and verifying that that they come in order) or on the kernel side force a non-diff every N frame (aka key-frames analogous to the way that mpeg works internally).

tacaswell · 2020-12-03T06:58:14Z

I also think than we can defer dealing with the key frames / generic skipping to a later PR.

thomasaarholt

Very excited to try this!

thomasaarholt · 2020-12-03T08:53:05Z

lib/matplotlib/backends/backend_webagg_core.py

-                                             dtype=np.uint32)
-                               .reshape((renderer.height, renderer.width)))
-                diff = buff != last_buffer
+                diff = buff != self._last_buff
                output = np.where(diff, buff, 0)

            buf = BytesIO()


I notice that there are both buff and buf in this code (I realise you didn't introduce or touch the last one). Do you think you could attempt to clear up what the difference between them is, and perhaps make the variable names self explanatory?

Making it clearer now will make it easier for the next developer who looks at this to understand what's going on :)

I haven't left a comment because I'm not 100% sure this is the reason. But here's the idea of what I might leave:

We differentiate between buff and buf because we cannot directly return buff as that is a reference to a location in memory that may change later.

There may also be something happening with the save where the buffer gets converted from whatever the renderer uses into a png buffer.

See #19117.

thomasaarholt · 2020-12-03T09:19:56Z

@ianhi I modified your top example to have a toggle for blitting (ExampleToggle(blit=False/True)), and added an output widget to track clicks on the figure (mostly for debugging). If the example is too slow for someone out there trying this, set IMSIZE = 1000 instead of 7000.

The speedup is very nice!

Code here

from imp import reload

import matplotlib
reload(matplotlib)

matplotlib.use('nbagg')

import matplotlib.backends.backend_nbagg
reload(matplotlib.backends.backend_nbagg)
import matplotlib.backends.backend_webagg_core
reload(matplotlib.backends.backend_webagg_core)

import numpy as np
import matplotlib.pyplot as plt
import matplotlib.patches as mpatches
from ipywidgets import Output
IMSIZE = 7000
RED = (1,0,0,1)
GREEN = (0,1,0,1)
o = Output()
plt.ion()
class ExampleToggle:
    
    def __init__(self, blit=True):
        self.blit = blit
        self.fig = plt.figure()
        self.ax = self.fig.add_subplot(111)
        self.canvas = self.ax.figure.canvas

        self.im = np.random.randint(0, 255, [IMSIZE,IMSIZE, 3]).astype('uint8')
        self.tmp_colour = RED
        self.rect = mpatches.Rectangle(
            [0, 0], 200, 200, 
            edgecolor='yellow', 
            facecolor=self.tmp_colour, 
            linewidth=0.5, 
            animated=self.blit)
        self.ax.add_patch(self.rect)
        
        self.canvas.mpl_connect('button_press_event', self.clicked)
        self.ax.imshow(self.im)
        self.canvas.draw()
        self._been_clicked = False

    def clicked(self, event):
        with o:
            print('clicked')
            if not self._been_clicked:
                self.background = self.canvas.copy_from_bbox(self.ax.bbox)    
                self._been_clicked = True

            if self.tmp_colour == RED:
                self.tmp_colour = GREEN
            elif self.tmp_colour == GREEN:
                self.tmp_colour = RED
            self.rect.set_facecolor(self.tmp_colour)
            if self.blit:
                self.canvas.restore_region(self.background)
                self.ax.draw_artist(self.rect)
                self.canvas.blit(self.ax.bbox)

et = ExampleToggle(blit=False)
display(o)

ianhi · 2020-12-03T16:53:10Z

lib/matplotlib/backends/backend_webagg_core.py

+    def blit(self, bbox=None):
+        self._png_is_old = True
+        self.manager.refresh_all()


I copied this from #4290 so I assume it was done correctly by @tacaswell, but I find it odd that this accepts a bbox argument but doesn't do anything with it. Is that a mistake?

That's to match the super class; it should blit only the section in the bbox if it's not None. Just re-blitting everything is fine, but perhaps a bit under-optimized. I think you would have to add to the current wire protocol to be able to do bbox'd blitting.

makes sense. Should a docstring be added explaining something to that effect?

I think I see how to implement more selective blitting for the ipympl frontend, but would/can the backend Agg renderer also needing blitting?

The other consideration is that if any part of the image is transparent it will be forced to be a full image. That could also be optimized a bit with frontend blitting. Discussed far in the past here: #5419 (comment)

I'm sure it could be possible, but would require sending a different message probably. Doesn't have to be done in this PR though.

Indeed I'm happy with this PR as is. Better blitting seems like a good 2021 goal.

QuLogic

Was that all that was needed? It seems so simple...

QuLogic · 2020-12-04T22:55:25Z

lib/matplotlib/backends/backend_webagg_core.py

+    def blit(self, bbox=None):
+        self._png_is_old = True
+        self.manager.refresh_all()


That's to match the super class; it should blit only the section in the bbox if it's not None. Just re-blitting everything is fine, but perhaps a bit under-optimized. I think you would have to add to the current wire protocol to be able to do bbox'd blitting.

lib/matplotlib/backends/web_backend/nbagg_uat.ipynb

…g figure taking the example from matplotlib#9240 Co-Authored-By: Thomas A Caswell <tcaswell@gmail.com>

As noted in matplotlib#19059, there is both a `buff` and `buf` in this function. The latter is a PNG file, so rename it to `png`.

meassinal · 2021-04-25T09:47:07Z

But I still see that supports_blit = False in the source code.

ianhi · 2021-04-25T16:11:50Z

@meassinal see: #19762 which was a follow up to this

meassinal · 2021-04-26T07:49:05Z

@ianhi thanks a lot for this. Anyway, I'm looking for an improvement regarding of performance issue of ax.imshow(img, origin='upper', interpolation="nearest") using backend_webagg_core along with tornado.web (websocket). I'm wondering when having animated artists interacted on the image caused slow performance, either in the matplotlib window but i could solve it with tkinter while in the web browser I couldn't. One way around is to use blit for redrawing speed but blit is not supportable in the webagg_core. I appreciate if you could give some tips regarding this issue.

ianhi · 2021-06-28T03:53:21Z

@meassinal sorry I missed this. If this is still an issue could you please ask about that on https://discourse.matplotlib.org/

ianhi mentioned this pull request Dec 3, 2020

blitting doesn't work - would be useful to speed up graph draw for faster realtime-graphs matplotlib/ipympl#228

Open

ianhi force-pushed the blit branch from 4f59d11 to c901b8d Compare December 3, 2020 04:12

tacaswell added this to the v3.4.0 milestone Dec 3, 2020

tacaswell added the status: needs review label Dec 3, 2020

tacaswell approved these changes Dec 3, 2020

View reviewed changes

thomasaarholt reviewed Dec 3, 2020

View reviewed changes

ianhi commented Dec 3, 2020

View reviewed changes

ianhi mentioned this pull request Dec 4, 2020

Use Blitting For all updates mpl-extensions/mpl-interactions#153

Open

QuLogic reviewed Dec 5, 2020

View reviewed changes

add blitting to nbagg UAT notebook + correct the numbering of stoppin…

2fbbec9

…g figure taking the example from matplotlib#9240 Co-Authored-By: Thomas A Caswell <tcaswell@gmail.com>

ianhi force-pushed the blit branch from c901b8d to 2fbbec9 Compare December 5, 2020 17:26

QuLogic approved these changes Dec 15, 2020

View reviewed changes

QuLogic merged commit 7db82ea into matplotlib:master Dec 15, 2020

ianhi deleted the blit branch December 15, 2020 04:53

ianhi mentioned this pull request Dec 15, 2020

Stale information drawn by nbAgg #19116

Open

QuLogic added a commit to QuLogic/matplotlib that referenced this pull request Dec 15, 2020

Rename a confusing variable.

4c5da38

As noted in matplotlib#19059, there is both a `buff` and `buf` in this function. The latter is a PNG file, so rename it to `png`.

QuLogic mentioned this pull request Dec 15, 2020

Rename a confusing variable. #19117

Merged

3 tasks

QuLogic removed the status: needs review label Mar 17, 2021

jdtsmith mentioned this pull request May 26, 2021

Fit Progress Visualizer PAHFIT/pahfit#81

Open

ianhi mentioned this pull request Jun 28, 2021

Matplotlib blitting in Jupyter Notebooks (issue #4288) #9240

Closed

anntzer mentioned this pull request Jun 1, 2022

Simplify webagg blitting. #23182

Merged

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support blitting in webagg backend #19059

Support blitting in webagg backend #19059

ianhi commented Dec 3, 2020 •

edited

ianhi commented Dec 3, 2020

tacaswell commented Dec 3, 2020

tacaswell commented Dec 3, 2020

thomasaarholt left a comment

thomasaarholt Dec 3, 2020 •

edited

ianhi Dec 5, 2020

QuLogic Dec 15, 2020

thomasaarholt commented Dec 3, 2020 •

edited

ianhi Dec 3, 2020

QuLogic Dec 4, 2020

ianhi Dec 5, 2020

ianhi Dec 5, 2020

ianhi Dec 5, 2020

QuLogic Dec 9, 2020

ianhi Dec 9, 2020

QuLogic left a comment

QuLogic Dec 4, 2020

meassinal commented Apr 25, 2021 •

edited

ianhi commented Apr 25, 2021

meassinal commented Apr 26, 2021

ianhi commented Jun 28, 2021

Support blitting in webagg backend #19059

Support blitting in webagg backend #19059

Conversation

ianhi commented Dec 3, 2020 • edited

PR Summary

PR Checklist

ianhi commented Dec 3, 2020

tacaswell commented Dec 3, 2020

tacaswell commented Dec 3, 2020

thomasaarholt left a comment

Choose a reason for hiding this comment

thomasaarholt Dec 3, 2020 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

thomasaarholt commented Dec 3, 2020 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

QuLogic left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

meassinal commented Apr 25, 2021 • edited

ianhi commented Apr 25, 2021

meassinal commented Apr 26, 2021

ianhi commented Jun 28, 2021

ianhi commented Dec 3, 2020 •

edited

thomasaarholt Dec 3, 2020 •

edited

thomasaarholt commented Dec 3, 2020 •

edited

meassinal commented Apr 25, 2021 •

edited