Nbagg backend

[pelson]

Implements the nbAgg backend for interactive figures in the IPython notebook, using IPython's comm protocol.

@jasongrout's excellent proof of concept CommFigure in #2524 was used as an initial base (which itself made good use of @mdboom and my own work on the WebAgg backend), but I've removed the CommFigure concept and have instead plumbed this into a real matplotlib backend.

As a result, using IPython notebook v2.0 I am able to run the following code in the notebook to achieve an interactive figure:

import matplotlib
matplotlib.use('nbagg')

import matplotlib.pyplot as plt
import numpy as np

x = np.linspace(0, 2 * np.pi, 1000)
plt.plot(x, np.cos(3*x) - np.sin(2*x)**2)
plt.show()

I've currently only been able to test this with Firefox 24.4, so any feedback from other browsers would be valuable.

There are a few features I've not added at this point (but would like to do so):

• An event loop
• Support for interactive mode and the ability to "re-show" a figure.
• Better keyboard event integration with IPython notebook (currently pressing the close button destroys all IPython keyboard events, and when the mpl figure has focus, no IPython shortcuts are triggered)
• Remove the hack which identifies which cell contains the output that has just been written (using the unique ID defined at show time of a figure).
• Adding %matplotlib nbagg (in IPython itself)
• Testing... (at least a user acceptance test script for individuals to run).

[jasongrout]

Cool! Thanks for taking this project up!

[mdboom]

It appears that (maybe) your editor borked the utf-8 here.

[mdboom]

Very cool! I can report that it works on Google Chrome 35 on Linux. It seems to be significantly slower for panning and zooming than the "raw" WebAgg backend. I wonder where the overheads are.

[pelson]

I wonder where the overheads are.

I've throttled the mouse events to 20ms, and there is no binary web sockets, so I suspect they are the cause. I've not tested this over a slow network - it might be that we need to do more to handle high latency connections.

[mdboom]

I should have mentioned -- the testing I did was locally on the same machine. We probably need to investigate dynamic throttling somehow based on the latency.

[ludwigschwardt]

Hi Phil,

This looks very promising! @sratcliffe and I plumbed our mplh5canvas backend into the notebook back in 2012 at EuroSciPy but that was before the comm protocol, which made it very hacky and we didn't take it further. This is definitely a cleaner approach.

I can confirm that nbagg works on Chrome 34 and Safari 7 on Mac OS X 10.9.2. Chrome has this funny issue that the antialiasing of the canvas lags any change to it by several seconds. So if you update the plot by zooming or panning, there is a slight loss of quality in the lines and text which corrects itself after up to 3 seconds (presumably by antialiasing kicking in). Safari seems snappier and does not have the antialiasing issue.

I can also confirm @mdboom's observation that panning is a bit laggy even when connected to localhost. When the pan cursor is let go, the plots shifts after maybe 100-250 ms. This is not as obvious with zooming, which does not require such high-speed updating as panning (i.e. the plot is not expected to move while you draw the reticule). I could also get the plot to become unresponsive by panning 3 or 4 times in quick succession (I have a knack for breaking plotting interfaces - ask @sratcliffe :-)).

Looking forward to the end result!

[dmcdougall]

@pelson This is fantastic work. Good job. This would be a great thing to announce and demo at SciPy this year.

[ellisonbg]

This is great a few comments though:

tl;dr: make the nbagg backend an IPython Widget rather than a Comm to enable users to hook up other widgets like sliders and checkboxes to their matplotlib plots and enable them to work on http://nbviewer.ipython.org.

Full explanation: I think that this should almost certainly be implemented as a Widget rather than a raw Comm. There are a number of advantages of Widgets for usage cases like this:

• You don't have to implement the extra logic that associated Comm objects with cells in the notebook.
• We are working on an architecture that will enable Widgets (but not Comms) to work on static web pages, like blogs and http://nbviewer.ipython.org.
• Widgets are reusable blocks that can be composed and nested in arbitrary ways.
• Widgets cleanly separate the stateful aspects from the message oriented aspects of UIs like this. While much of the nbagg backend is message oriented, there are definitely some aspects that are stateful. To manage that state you have to build state management on top of the Comm layer - Widgets already do that.
• Widgets have a clearly defined and user-focused event model that can be used to tie different widgets together. This allows for very sophisticated behaviors in systems of multiple widgets.
• Widgets have a clearly defined place in each cell for their display.
• Widgets have automated lifecycle management that takes care of synchronizing multiple instances of the same widget in different cells - even if those widgets have different parents in each cell.
• We are also working to make sure that Widgets will persist across page refresh/reloads.
• Widgets work with the higher level interact/interactive APIs.
• There should be no significant performance different between the message oriented API for Widgets and the message oriented API for Comms.

[jasongrout]

I agree with @ellisonbg that using the widget architecture is something to be seriously considered. I think "widgets have a clearly defined place in each cell for their display" is a disadvantage, not an advantage, but other than that, I think his list of advantages is pretty good.

[ellisonbg]

Warning: ongoing subtle design discussion about widgets/comms:

Keep in mind though @jasongrout - even if we start to treat widgets like regular output and allow them to be interleaved with regular output - widgets will still have "a clearly defined place in each cell for their display" - there will just be multiple of them with the same CSS class, UI, etc.

The important thing is that Comms' have absolutely no clear place in a cell for visible content. If you want to put content on the page related to aCommyou have to literally make an arbitrary choice about which DOM parent to use, which CSS classes, to use, etc. You could pick the entire page, the cell div, the output div, etc. If a Comm appends its content inside a cell's div, it has to watch the cell careful to see if it is deleted or changed to the markdown/heading cell, so the Comm's display logic can follow. Another way of saying this is that theCommlayer is only about sending messages, it is most definitely not an MVC system and especially has absolutely no concept of a view. You will end up inventing your own MVC from scratch if you use rawComm`s.

However the question of how widgets relate to and interleave with regular output is something that we are thinking about in the background. We don't plan on making any changes to how that works soon, but we realize the limitations of our current approach of one widget area per cell that is separate from output.

[jasongrout]

Warning: this continues the subtle discussion @ellisonbg mentions:

You said: widgets will still have "a clearly defined place in each cell for their display" - there will just be multiple of them with the same CSS class, UI, etc. So then they won't have a place, but it might be more correct to say widgets will still have clearly defined, managed places in each cell for their display -- I'm happy to count that as an advantage! And I agree that the automatic management of output regions is a big plus in general.

(sorry; that's probably just the technical, picky, mathematician part of me taking apart the grammar to extract exact meaning...).

[ellisonbg]

I completely agree - the "a" in my original statement was a bit ambiguous.

[pelson]

Thanks for the feedback @ellisonbg - the only advantage I needed to read was:

• You don't have to implement the extra logic that associated Comm objects with cells in the notebook.

And you had me convinced. Whilst clearly I've been able to reverse engineer a solution based on a unique div id, I fully accept it is a hack that I'd love to get rid of, so if the Widget infrastructure can help in that regard, I'm all for it.

Out of interest, did I miss anything with regards to the save event? It really does look like the notebook_saving.Notebook event is fired after the notebook is JSONised, which is too late for the interactive figure to push a static image into the saved notebook. Does the Widget infrastructure help in this regard?

Not having read up on the new widgeting system, my only slight concern is the ability to hook into the existing JS interface construction that I've inherited from the WebAgg backend, but that really is much of a muchness, and in truth is only about 30 lines of JS.

@mdboom et al. - I think there are some things in here we want to keep, and some things in here which will be superseded once we make use of the Widget infrastructure. As such I suggest we merge this as an experimental feature and iterate on what is here, rather than rejecting outright. Of course, I would say that... so does anybody else have a view on this?

[jasongrout]

If you're trying to iterate, I think you ought to look at the custom message handlers that widgets provide. They basically provide a way for the python and javascript sides of a widget to communicate. You can probably basically swap in the current infrastructure into a widget.

Basically, you'll need to create a new view class with a render function that renders the matplotlib image to the this.el or this.$el element (this.$el is just a jquery version of the element). Then you can use custom messages handlers to send messages back and forth.

Long-term, breaking out the parts of the interface that are nicely represented as state that needs to be synced between python and javascript and putting that into the model is the way to go, I think.

[rossant]

We are very interested in this widget-based approach for Vispy, which would provide an interactive web backend in the IPython notebook. I'm wondering whether it would be sensible to standardize a bit the custom messages sent between Python and Javascript. Things like mouse actions, key pressed, etc. on the one hand, and images on the other hand, are likely to be similar between matplotlib, vispy and other visualization projects.

We're also considering another type of backend where we'd not stream PNG images from Python to Javascript, but raw WebGL commands.

[mdboom]

I'm fine with merging this and iterating toward the widget-based approach.

It does seem that this is creates a performance regression -- even for the "raw" WebAgg backend -- and I'd like to at least resolve that before merging. I think the event throttling is maybe not the way to go. I think instead we need to do event queuing such that intermediate mouse moves can be ignored if the updates get behind. That way we always get maximum throughput rather than setting an arbitrary upper limit on performance. What happens if we just take the throttling out for now, and then do something more adaptive in a subsequent PR?

[jasongrout]

Right now, the state of the widget in syncing is throttled, but state changes are merged. By default, IIRC, it allows 3 state change messages out. Any more state changes than that (for example, rapidly moving mouse) get merged into a single state change that is sent out. So if you set the throttle to 1, and use the state syncing to sync a current mouse position between python and javascript, then I think you can have your queuing for free.

I would be interested in performance regressions with the widget system. It seems like there might be some, especially compared to the original webagg because we can't use binary web sockets with the widgets.

[pelson]

What happens if we just take the throttling out for now, and then do something more adaptive in a subsequent PR?

Sounds reasonable.

I'll get onto rebasing this and removing the throttling.

[Tillsten]

@tacaswell Getting this into 1.4 would be very nice!

[tacaswell]

@person can I delegate the triageing if this for 1.4.0 to you?

[tacaswell]

Tagging an 1.4.0, we can always punt later.

[pelson]

I've finally got around to updating this PR. Thanks for the nudge @tacaswell.
The throttling has now been removed, and some behavioural tweaks have been made to make draw/show behaviour more consistent with other backends. I'm still none the wiser how some of the show/interactive machinery works after all these years - clearly my mental model of what is going on is slightly out of kilt - perhaps we should look at documenting this, as I suspect there are very few people who do. That may be one for a face-to-face discussion with @mdboom at SciPy.

Other than that, I think this is good to go.

[tacaswell]

Dealt with the conflict and merged.

[pelson]

Thanks for merging @tacaswell. Good to have this in before SciPy'14! 👍

[JohnGriffiths]

Just discovered this. It's a really great development. Quick suggestion: adding some kind of 'snapshot' button that e.g. saves to .png would be extremely handy for the kind of in-notebook fast data exploration + prototyping that this is presumably intended to facilitate. Load up interactive plot, explore, change parameters, see something interesting; save snapshot; carry on exploring.

[tacaswell]

Can you expand a bit on what you want? There is currently a x at the top
right which will kill the inactivity and turn the figure in to a fixed png.

Do you want a download option or the ability to create a gallery on the fly?

On Tue, Nov 18, 2014, 20:16 John Griffiths notifications@github.com wrote:

Just discovered this. It's a really great development. Quick suggestion:
adding some kind of 'snapshot' button that e.g. saves to .png would be
extremely handy for the kind of in-notebook fast data exploration +
prototyping that this is presumably intended to facilitate. Load up
interactive plot, explore, change parameters, see something interesting;
save snapshot; carry on exploring.

—
Reply to this email directly or view it on GitHub
#3008 (comment)
.

[WeatherGod]

"an 'x' at the top right" -- right, because the first thing I think when I
want to save a figure is to click on the 'x' in the corner...

On Tue, Nov 18, 2014 at 8:19 PM, Thomas A Caswell notifications@github.com
wrote:

Can you expand a bit on what you want? There is currently a x at the top
right which will kill the inactivity and turn the figure in to a fixed
png.

Do you want a download option or the ability to create a gallery on the
fly?

On Tue, Nov 18, 2014, 20:16 John Griffiths notifications@github.com
wrote:

Just discovered this. It's a really great development. Quick suggestion:
adding some kind of 'snapshot' button that e.g. saves to .png would be
extremely handy for the kind of in-notebook fast data exploration +
prototyping that this is presumably intended to facilitate. Load up
interactive plot, explore, change parameters, see something interesting;
save snapshot; carry on exploring.

—
Reply to this email directly or view it on GitHub
<
https://github.com/matplotlib/matplotlib/pull/3008#issuecomment-63576834>
.

—
Reply to this email directly or view it on GitHub
#3008 (comment)
.

[mdboom]

The underlying WebAgg control has an option to save the file, but it seems NbAgg doesn't. (I'm surprised I'm just noticing that). @pelson: Any reason why the save dropdown isn't available for NbAgg?

[JohnGriffiths]

Thanks, didn't know about the 'x'. That does help.

In general I prefer to be able to save important figures to separate files though; keeping them squirreled away in the notebook can be a little restrictive.

I played around a bit and a reasonable working solution seems to be to simply define a function like

def save_and_show_fig(fname):
  fig.canvas.print_figure(fname)
  return Image(fname)

At least for the slider demo, this allows me to play around with the figure interactively and output the current state to separate cells as desired, as well as saving to file.

Incidentally, I also tried adding a 'save button' to the slider example

saveax = plt.axes([0.2, 0.025, 0.1, 0.04])
savebutton = Button(saveax, 'Save', color=axcolor, hovercolor='0.975')
fignum = 1
def savepng(event):
    fig.canvas.print_png('/tmp/savefig_%s.png' %fignum)    
    fignum+=1
savebutton.on_clicked(savepng)

...but this didn't work.

In any case the simpler option above is probably more useful as does both the saving and the printing of static images to new cells.

[tacaswell]

@JohnGriffiths Can you make a new issue summarizing the current state of this?

It will make it easier for us to keep track of it.

[pelson]

@mdboom - no major reason. The save button would be a reasonable extension, I seem to remember not knowing what end point to use (i.e. how to map a URL that can actually be downloaded in the IPython notebook's domain).

@JohnGriffiths - at one point, I considered the possibility allowing users to "snapshot" an image, carry on interacting, "snapshot" another, and be able to look at each of the snapshots in the resulting (static) notebook, but it sounds like this isn't really what you want. Instead, you just want to save the figure to disk, right? FWIW, if you have a reference to the figure, the nbagg backend is not blocking, so you can always simply do fig.savefig(...) to save the current state.

[JohnGriffiths]

Yes, the snapshot saving is the primary thing I'm interested in here. Saving to disk would be a separate but complementary feature. Perhaps it could be a configurable option in the snapshots.

Regarding the snapshots: it would useful to output these in some kind of 'gallery viewer' - i.e. a single cell which you can click-and-browse through the selected snapshots.

A very simple version of this using static interactive widgets from ipywidgets would be:

from ipywidgets import StaticInteract, DropDownWidget
from IPython.display import Image

imgs = <list of .png files>
labs = <list of brief labels>


def show_static_img(img):   
  i = Image(img)
  return i

StaticInteract(show_static_img,
                       img = DropDownWidget(imgs),
                       labels=labs)

This gives you a cell with a drop-down menu for looking at the saved snapshots, which works in static views with nbviewer. I'm sure this could be made a lot slicker.

Some version of a single-cell snapshot gallery like this would IMHO be the perfect complement to the nbagg backend; dramatically increase its usefulness.