Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

WebAgg backend #1426

Merged
merged 18 commits into from

6 participants

Michael Droettboom Phil Elson Jens Hedegaard Nielsen Benjamin Root Cameron Bates Hans Meine
Michael Droettboom
Owner

This adds a WebAgg backend based on the approach I described here:

http://mdboom.github.com/blog/2012/10/11/matplotlib-in-the-browser-its-coming/

It's not complete, but I thought I would post this early to give it lots of browser testing and make sure that the fundamental approach is sound.

This does not integrate with IPython (though that's the plan) -- it is just a standalone backend, and the only additional dependency is Tornado.

To try it out, set your backend to WebAgg, make a figure and call pyplot.show(). By default, a webbrowser tab will be opened showing the figure (though the webbrowser behavior is configurable). Some fun examples to try are examples/animation/double_pendulum_animated.py and examples/event_handling/lasso.py.

Each of the existing figures is available on its own page (just like their own window in a GUI backend), and the URLs are of the form:

http://127.0.0.1:8888/1/
http://127.0.0.1:8888/2/
etc...

I don't have a lot of experience with browser programming, so the CSS and layout is a little clunky. I'd encourage improvements there, as well as any reports of any examples that don't seem to work.

Michael Droettboom
Owner

I should add that the "Zoom to Rect" functionality isn't drawing a rectangle yet, but that's coming.

lib/matplotlib/animation.py
@@ -500,7 +500,7 @@ class Animation(object):
'''
def __init__(self, fig, event_source=None, blit=False):
self._fig = fig
- self._blit = blit
+ self._blit = blit and fig.canvas.supports_blit
Phil Elson Collaborator
pelson added a note

This seems a little unfriendly. If I ask for blitting, I would expect to get it - otherwise I should get an error. What do you think?

Michael Droettboom Owner
mdboom added a note

This deserves some explanation. Some backends, such as WebAgg, do not support blitting. In the case of WebAgg, it's because blitting is not possible -- we don't have direct access to the window/screen buffers anyway, so there is no advantage to it. This allows code that was written to expect blitting (such as all of the examples/animation examples) to still work. There should be consistent behavior across backends, even when certain features are not available.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Phil Elson pelson commented on the diff
lib/matplotlib/backend_bases.py
@@ -1476,6 +1476,8 @@ class FigureCanvasBase(object):
'close_event'
]
+ supports_blit = True
Phil Elson Collaborator
pelson added a note

I wonder if it should be the other way around: By default a backend cannot blit, but if you implement the functionality, and toggle the switch, you get it...

Michael Droettboom Owner
mdboom added a note

That's possible -- I was just trying to be backward compatible here with third-party backends that don't currently implement this value.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
lib/matplotlib/backends/backend_webagg.py
((92 lines not shown))
+ self._timer.stop()
+ self._timer = None
+
+ def _timer_set_interval(self):
+ # Only stop and restart it if the timer has already been started
+ if self._timer is not None:
+ self._timer_stop()
+ self._timer_start()
+
+
+class FigureCanvasWebAgg(backend_agg.FigureCanvasAgg):
+ supports_blit = False
+
+ def __init__(self, *args, **kwargs):
+ backend_agg.FigureCanvasAgg.__init__(self, *args, **kwargs)
+ self.png_buffer = cStringIO.StringIO()
Phil Elson Collaborator
pelson added a note

It would be great to get some docstrings on these attributes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
lib/matplotlib/backends/backend_webagg.py
((105 lines not shown))
+ def __init__(self, *args, **kwargs):
+ backend_agg.FigureCanvasAgg.__init__(self, *args, **kwargs)
+ self.png_buffer = cStringIO.StringIO()
+ self.png_is_old = True
+ self.force_full = True
+ self.pending_draw = None
+
+ def show(self):
+ # show the figure window
+ show()
+
+ def draw(self):
+ # TODO: Do we just queue the drawing here? That's what Gtk does
+ renderer = self.get_renderer()
+
+ self.png_is_old = True
Phil Elson Collaborator
pelson added a note

I think this is probably a private attribute (users shouldn't fiddle with it).

Michael Droettboom Owner
mdboom added a note

Agreed. Will fix.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
lib/matplotlib/backends/backend_webagg.py
((131 lines not shown))
+ if self.pending_draw is None:
+ ioloop = tornado.ioloop.IOLoop.instance()
+ self.pending_draw = ioloop.add_timeout(
+ datetime.timedelta(milliseconds=50),
+ self._draw_idle_callback)
+
+ def _draw_idle_callback(self):
+ try:
+ self.draw()
+ finally:
+ self.pending_draw = None
+
+ def get_diff_image(self):
+ if self.png_is_old:
+ buffer = np.frombuffer(
+ self.renderer.buffer_rgba(), dtype=np.uint32)
Phil Elson Collaborator
pelson added a note

Are these need to be 32 bit integers and not uint8?

Michael Droettboom Owner
mdboom added a note

Probably deserving of a comment. This makes things much faster for the difference comparison, because one can then say buffer != last_buffer rather than buffer[...:0] != last_buffer[...:0] and buffer[...:1] != last_buffer[...:1] etc..

Phil Elson Collaborator
pelson added a note

Aha. I see the next line does buffer.reshape((self.renderer.height, self.renderer.width)) not buffer.reshape((self.renderer.height, self.renderer.width, 4)). I see your motivation now :smile: .

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
lib/matplotlib/backends/backend_webagg.py
((132 lines not shown))
+ ioloop = tornado.ioloop.IOLoop.instance()
+ self.pending_draw = ioloop.add_timeout(
+ datetime.timedelta(milliseconds=50),
+ self._draw_idle_callback)
+
+ def _draw_idle_callback(self):
+ try:
+ self.draw()
+ finally:
+ self.pending_draw = None
+
+ def get_diff_image(self):
+ if self.png_is_old:
+ buffer = np.frombuffer(
+ self.renderer.buffer_rgba(), dtype=np.uint32)
+ buffer = buffer.reshape(
Phil Elson Collaborator
pelson added a note

The numpy docs suggest that if you definitely don't want to copy the data you should modify the shape attribute: http://docs.scipy.org/doc/numpy/reference/generated/numpy.reshape.html

Hans Meine
hmeine added a note

Interesting, I did not know that reshape could lead to a copy. However, assigning to .shape does make the code more ugly in many cases (because it cannot be chained), and the caveat is not relevant here, since it only applies to cases where ("incompatible") dimensions are reduced/joined, not split.

Phil Elson Collaborator
pelson added a note

more ugly in many cases (because it cannot be chained)

I agree completely with what you are saying:

data = numpy.arange(12).reshape((3, 4))

is better than

data = numpy.arange(12)
data.shape = (3, 4)

But in general, I find reading highly chained code very hard to read (d3.js being the very obvious example).

In this case, despite the fact that it is possible to chain with reshape, it hasn't actually been used. So whenever I see

data = data.reshape(new_shape)

I would tend to write:

data = data.shape = new_shape

Cheers! :smiley:

Michael Droettboom Owner
mdboom added a note

Good point. I agree with both of these comments, but I think in the end it probably makes sense to assign to shape so that we'll get an exception if we break something in the future where the dimensions in fact don't match.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
lib/matplotlib/backends/backend_webagg.py
((142 lines not shown))
+
+ def get_diff_image(self):
+ if self.png_is_old:
+ buffer = np.frombuffer(
+ self.renderer.buffer_rgba(), dtype=np.uint32)
+ buffer = buffer.reshape(
+ (self.renderer.height, self.renderer.width))
+
+ if not self.force_full:
+ last_buffer = np.frombuffer(
+ self.last_renderer.buffer_rgba(), dtype=np.uint32)
+ last_buffer = last_buffer.reshape(
+ (self.renderer.height, self.renderer.width))
+
+ diff = buffer != last_buffer
+ output = np.where(diff, buffer, 0)
Phil Elson Collaborator
pelson added a note

I'm not sure if it is more expensive, but I find these two lines easier to read in the following form:

Old:

diff = a % 2 == 0 
output = np.where(diff, a, 0)

New:

a[a % 2 != 0] = 0

Obviously they have different semantics, but given the output array is only in scope in this function, it seems reasonable to modify it inplace...

Michael Droettboom Owner
mdboom added a note

The inplace (what you're suggesting) is probably less expensive. I'll experiment with that.

Michael Droettboom Owner
mdboom added a note

Unfortunately inplace won't work here because we can remove pixels from the original image, or further differencing will fail.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
lib/matplotlib/backends/backend_webagg.py
((135 lines not shown))
+ self._draw_idle_callback)
+
+ def _draw_idle_callback(self):
+ try:
+ self.draw()
+ finally:
+ self.pending_draw = None
+
+ def get_diff_image(self):
+ if self.png_is_old:
+ buffer = np.frombuffer(
+ self.renderer.buffer_rgba(), dtype=np.uint32)
+ buffer = buffer.reshape(
+ (self.renderer.height, self.renderer.width))
+
+ if not self.force_full:
Phil Elson Collaborator
pelson added a note

Presumably this is currently always true as I can't see any code further on to handle image diffs vs actual images? Is that correct?

Phil Elson Collaborator
pelson added a note

Ah. It seems not... ok, that bit was a little confusing, perhaps an overall comment describing the strategy of this method would be helpful here.

Michael Droettboom Owner
mdboom added a note

Whenever a new client comes on line, it needs to get a "full" over the socket to start with. It does this by sending a "refresh" event to the server. I agree this deserves a comment.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
lib/matplotlib/backends/backend_webagg.py
((146 lines not shown))
+ self.renderer.buffer_rgba(), dtype=np.uint32)
+ buffer = buffer.reshape(
+ (self.renderer.height, self.renderer.width))
+
+ if not self.force_full:
+ last_buffer = np.frombuffer(
+ self.last_renderer.buffer_rgba(), dtype=np.uint32)
+ last_buffer = last_buffer.reshape(
+ (self.renderer.height, self.renderer.width))
+
+ diff = buffer != last_buffer
+ output = np.where(diff, buffer, 0)
+ else:
+ output = buffer
+
+ self.png_buffer.reset()
Phil Elson Collaborator
pelson added a note

I think these two lines probably need a comment...

Michael Droettboom Owner
mdboom added a note

Sure. The gist is that this prevents deallocating/reallocating a StringIO buffer on every call.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
lib/matplotlib/backends/backend_webagg.py
((189 lines not shown))
+ self._lastKey = key
+
+ return self.renderer
+
+ def handle_event(self, event):
+ type = event['type']
+ if type in ('button_press', 'button_release', 'motion_notify'):
+ x = event['x']
+ y = event['y']
+ y = self.get_renderer().height - y
+
+ # Javascript button numbers and matplotlib button numbers are
+ # off by 1
+ button = event['button'] + 1
+
+ # The right mouse button pops up a context menu, which doesn't
Phil Elson Collaborator
pelson added a note

We need a javascript guru - the context menu is definately disable-able...

Michael Droettboom Owner
mdboom added a note

I found a few solutions, but none seemed to work with Google Chrome 23. My understanding is that Chrome no longer lets Javascript do this for understandable security reasons.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
lib/matplotlib/backends/backend_webagg.py
((237 lines not shown))
+ self, timeout)
+ start_event_loop.__doc__ = backend_bases.FigureCanvasBase.start_event_loop_default.__doc__
+
+ def stop_event_loop(self):
+ backend_bases.FigureCanvasBase.stop_event_loop_default(self)
+ stop_event_loop.__doc__ = backend_bases.FigureCanvasBase.stop_event_loop_default.__doc__
+
+
+class FigureManagerWebAgg(backend_bases.FigureManagerBase):
+ def __init__(self, canvas, num):
+ backend_bases.FigureManagerBase.__init__(self, canvas, num)
+
+ self.web_sockets = set()
+
+ self.canvas = canvas
+ self.num = num
Phil Elson Collaborator
pelson added a note

These two are already handled by the superclass.

Michael Droettboom Owner
mdboom added a note

Good catch. Will fix.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
lib/matplotlib/backends/backend_webagg.py
((267 lines not shown))
+ s.send_image()
+
+ def send_event(self, event_type, **kwargs):
+ for s in self.web_sockets:
+ s.send_event(event_type, **kwargs)
+
+ def _get_toolbar(self, canvas):
+ toolbar = NavigationToolbar2WebAgg(canvas)
+ return toolbar
+
+ def resize(self, w, h):
+ self.send_event('resize', size=(w, h))
+
+
+class NavigationToolbar2WebAgg(backend_bases.NavigationToolbar2):
+ toolitems = (
Phil Elson Collaborator
pelson added a note

These are available from the superclass. I'm not sure if its better or not to do:

toolitems = NavigationToolbar2.toolitems[:-2] + ('Download', 'Download plot', 'filesave', 'download')
Michael Droettboom Owner
mdboom added a note

I like the idea of being explicit here. There will always be some options that don't make sense in this context -- the "configure subplots" button, and the "save" button which needs to behave differently. I suppose we could always just copy the first few, but I was worried that would become brittle if the standard GUI set of buttons ever changed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Phil Elson pelson commented on the diff
lib/matplotlib/backends/backend_webagg.py
((307 lines not shown))
+ self.canvas.send_event("cursor", cursor=cursor)
+ self.cursor = cursor
+
+ def dynamic_update(self):
+ self.canvas.draw_idle()
+
+
+class WebAggApplication(tornado.web.Application):
+ initialized = False
+ started = False
+
+ class FavIcon(tornado.web.RequestHandler):
+ def get(self):
+ self.set_header('Content-Type', 'image/png')
+ with open(os.path.join(
+ os.path.dirname(__file__),
Phil Elson Collaborator
pelson added a note

I think there is an rcParam for that....

Michael Droettboom Owner
mdboom added a note

For the icon image?

Phil Elson Collaborator
pelson added a note

No. Apologies, I meant to get to the mpl-data directory (the subject of which is in another comment later on).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
lib/matplotlib/backends/backend_webagg.py
((345 lines not shown))
+
+ # TODO: Move this to a central location
+ mimetypes = {
+ 'ps': 'application/postscript',
+ 'eps': 'application/postscript',
+ 'pdf': 'application/pdf',
+ 'svg': 'image/svg+xml',
+ 'png': 'image/png',
+ 'jpeg': 'image/jpeg',
+ 'tif': 'image/tiff',
+ 'emf': 'application/emf'
+ }
+
+ self.set_header('Content-Type', mimetypes.get(format, 'binary'))
+
+ buffer = cStringIO.StringIO()
Phil Elson Collaborator
pelson added a note

Thinking about it, it would be sensible to use the python3 compatible StringIO here...

Michael Droettboom Owner
mdboom added a note

Agreed. Should be BytesIO, though.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
lib/matplotlib/backends/backend_webagg.py
((352 lines not shown))
+ 'png': 'image/png',
+ 'jpeg': 'image/jpeg',
+ 'tif': 'image/tiff',
+ 'emf': 'application/emf'
+ }
+
+ self.set_header('Content-Type', mimetypes.get(format, 'binary'))
+
+ buffer = cStringIO.StringIO()
+ manager.canvas.print_figure(buffer, format=format)
+ self.write(buffer.getvalue())
+
+ class WebSocket(tornado.websocket.WebSocketHandler):
+ def open(self, fignum):
+ self.fignum = int(fignum)
+ manager = Gcf().get_fig_manager(self.fignum)
Phil Elson Collaborator
pelson added a note

Gcf is a singleton (actually it is simply a namespace...). Creating an instance of it does nothing (and should probably raise an exception when trying to instantiate one).

Michael Droettboom Owner
mdboom added a note

Sure. Will fix.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
lib/matplotlib/backends/backend_webagg.py
((418 lines not shown))
+
+ def random_ports(port, n):
+ """Generate a list of n random ports near the given port.
+
+ The first 5 ports will be sequential, and the remaining n-5 will be
+ randomly selected in the range [port-2*n, port+2*n].
+ """
+ for i in range(min(5, n)):
+ yield port + i
+ for i in range(n - 5):
+ yield port + random.randint(-2 * n, 2 * n)
+
+ success = None
+ cls.port = rcParams['webagg.port']
+ # TODO: Configure port_retrues
+ for port in random_ports(cls.port, 50):
Phil Elson Collaborator
pelson added a note

This block frightens me a little, probably as I don't know tornado or much about web sockets. What is significant about the number of attempts? How does one know which port to use in the web browser URL (if you have to type it yourself rather than letting it get shown automatically)?

Michael Droettboom Owner
mdboom added a note

I borrowed this strategy from IPython notebook -- I should add a comment to that effect. I don't think there's a strategy to the number of attempts, but I will expose it as an rcParam (as IPython does).

Agreed - it needs to display the port that it arrived at. To be more friendly, it should probably display the entire URL, which in many Terminal emulators will automatically become clickable.

Phil Elson Collaborator
pelson added a note

How does one know which port to use in the web browser URL (if you have to type it yourself rather than letting it get shown automatically)?

I think this is a stupid question... these ports are for the websockets stuff right?

Michael Droettboom Owner
mdboom added a note

These ports are actually for the entire web server that Tornado creates, including the main web page and the static content.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
lib/matplotlib/backends/web_static/index.html
@@ -0,0 +1,44 @@
+<html>
+ <head>
+ <script src="/static/mpl.js"></script>
+ <link href="/static/mpl.css" rel="stylesheet" type="text/css">
+ </head>
+ <body
+ onkeydown="key_event(event, 'key_press')"
+ onkeyup="key_event(event, 'key_release')">
+ <div id="mpl-div">
+ <canvas id="mpl-canvas" width="800" height="600"
Phil Elson Collaborator
pelson added a note

We may as well do the width & height in the CSS... or does that break the resizing code?

Michael Droettboom Owner
mdboom added a note

I'll try that -- I don't know if it will break or not.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
lib/matplotlib/rcsetup.py
@@ -20,13 +20,13 @@
from matplotlib.colors import is_color_like
#interactive_bk = ['gtk', 'gtkagg', 'gtkcairo', 'fltkagg', 'qtagg', 'qt4agg',
-# 'tkagg', 'wx', 'wxagg', 'cocoaagg']
+# 'tkagg', 'wx', 'wxagg', 'cocoaagg', 'web']
Phil Elson Collaborator
pelson added a note

Not sure about this addition. Obviously it has no impact at this stage, but do we really want the lower case version to be different to the upper case one?

Michael Droettboom Owner
mdboom added a note

That's just a typo -- will fix to match the other.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
setup.py
@@ -107,6 +107,7 @@
'mpl-data/sample_data/*.*',
'mpl-data/sample_data/axes_grid/*.*',
'backends/Matplotlib.nib/*',
+ 'backends/web_static/*'
Phil Elson Collaborator
pelson added a note

I wonder if this stuff should live in mpl-data - given the fact that the mpl-data folder has a subfolder for sample_data, mpl-data could very easily have a web_static directory...

Michael Droettboom Owner
mdboom added a note

Yeah -- I waffled about that. There is other backend-specific data here in Matplotlib.nib, and since this is backend-specific I put it here. But it's not something I feel too strongly about.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Phil Elson
Collaborator

Note: Tornado should probably get a mention in the setup "optional dependencies" list.

Phil Elson pelson commented on the diff
lib/matplotlib/backends/backend_webagg.py
((88 lines not shown))
+ self._timer.start()
+
+ def _timer_stop(self):
+ if self._timer is not None:
+ self._timer.stop()
+ self._timer = None
+
+ def _timer_set_interval(self):
+ # Only stop and restart it if the timer has already been started
+ if self._timer is not None:
+ self._timer_stop()
+ self._timer_start()
+
+
+class FigureCanvasWebAgg(backend_agg.FigureCanvasAgg):
+ supports_blit = False
Phil Elson Collaborator
pelson added a note

As far as I can tell, this isn't a fundamental property of this approach (i.e. in the future, we will be able to blit). Is that correct?

Michael Droettboom Owner
mdboom added a note

No -- there is no way to blit directly to the screen/window hardware so there is no advantage to that approach. It could be faked, but I'm not sure why...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Phil Elson
Collaborator

Sadly, using web sockets means I can't use this backend with Firefox 10 (released January 31, 2012) - see http://en.wikipedia.org/wiki/WebSocket#Browser_support

I was able to test your previous tech demo, so I wonder if we can provide a fallback capability to less able browsers?

lib/matplotlib/backends/backend_webagg.py
((377 lines not shown))
+ message = json.loads(message)
+ canvas = Gcf().get_fig_manager(self.fignum).canvas
+ canvas.handle_event(message)
+
+ def send_event(self, event_type, **kwargs):
+ payload = {'type': event_type}
+ payload.update(kwargs)
+ self.write_message(json.dumps(payload))
+
+ def send_image(self):
+ canvas = Gcf().get_fig_manager(self.fignum).canvas
+ diff = canvas.get_diff_image()
+ self.write_message(diff, binary=True)
+
+ def __init__(self):
+ super(WebAggApplication, self).__init__([
Phil Elson Collaborator
pelson added a note

We should add an index page (and maybe a 404 error page), which lists all of the available figures (for bonus points, it would be nice to use the figure.get_label for that).

Michael Droettboom Owner
mdboom added a note

Sure. I planned to do this, but forgot to mention it.

Michael Droettboom Owner
mdboom added a note

My plan here is to provide a page with "thumbnails" of all of the open figures (with the figure.get_label beneath it).

Phil Elson Collaborator
pelson added a note

Nice idea! Extra bonus points for that! :smile:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Phil Elson
Collaborator

Mike: This is awesome stuff! The approach you have taken here is pragmatic in that it achieves 80% of what is really required (interactive figures in the web browser which are easily shareable to local users). Obviously, the approach falls short of being able to have the figures disconnected from a python/matplotlib server and globally shareable (securely), but realistically that requirement can only be met by re-writing large chunks of matplotlib in JavaScript which really does not sit well for a community driven Python package.

I've left quite a few comments: I hope you find them useful - the majority of them are relating to documentation/commenting. Other than that, I would be in favour of merging this early and labelling it an experimental feature (and getting it out for users to try in 1.3.x).

Great stuff!

Michael Droettboom
Owner

I hadn't considered the minimum requirements for WebSockets. Unfortunately, the approach used in the original tech demo made sharing these plots a lot more complicated -- since there was no way to "push" images to the browser, the browsers had to poll occasionally, and to stay in sync (i.e. to not drop frames) full frames would have to be pushed on polling. This approach with WebSockets is a lot cleaner and less likely to fail. But sure, if supporting the older browsers is deemed important, we do have a possible solution for a backward compatibility mode -- but I'd prefer not to have the complication if we can avoid it.

I should also add that the IPython notebook requires websockets, so there is incentive on a number of fronts for users to need a more recent browser.

Jens Hedegaard Nielsen

Note that IPython is considering to replace pure websockets with SockJS which should increase the browser coverage: ipython/ipython#2321

Michael Droettboom
Owner

@jenshnielsen: Thanks for the pointer. I'll look into that.

Michael Droettboom mdboom referenced this pull request from a commit in mdboom/matplotlib
Michael Droettboom mdboom Address documentation and smaller issues raised by @pelson in #1426. 7af2b90
Michael Droettboom
Owner

Regarding WebSocket browser coverage: It is specifically the binary websocket messages (that are used for the image data) that don't work with Firefox < 11. The text-based messages appear to work fine, with a simple Firefox compatibility hack.

SockJS looks like a nice solution, and the built-in support for broadcasting messages is a nice convenience we could use here. However, it doesn't yet support binary messages, so it doesn't really address our needs. Even if it does support binary messages, I'm not sure there's any way to do it without base64 encoding, which will probably add a lot of overhead. I'm also not too keen, as IPython has done, or including a bunch of external code to make this work... (There might be a way to get pip to require it etc.) I don't want to grow a dependency on IPython for this basic usage, either (nothing wrong with IPython, but I do like that it is easy/simple to get this going without it).

Anyway, I haven't decided either way yet about using SockJS -- that will require some experimentation.

Michael Droettboom mdboom closed this
Michael Droettboom mdboom reopened this
Phil Elson
Collaborator

I tested this on Firefox 14 with the lasso example. The lasso selector works great, but none of the toolbar buttons are working (even the download one). The only exception I can find is in the firefox exception console:

Timestamp: 23/10/2012 21:12:05
Error: The connection to ws://127.0.0.1:8888/1/ws was interrupted while the page was loading.
Source File: http://127.0.0.1:8888/static/mpl.js
Line: 29

This is using:

git log -n 1
commit 033ffb749450b713525b5905d2319ea9f14b8926
Author: Michael Droettboom <mdboom@gmail.com>
Date:   Tue Oct 23 13:49:03 2012 -0400

    Display URL if not set to open webbrowser

I don't know if this is because the backend opens up a window in safari, which I closed and then opened one in Firefox, or if that is a complete red-herring...

Michael Droettboom
Owner

@pelson: I just tried with that version of Firefox 14, and I think I now have an approach that works to get those toolbar button events. Browser programming seems to be a game of Whak-A-Mole ;)

Benjamin Root
Collaborator

Another data point: the double-pendulum example works on CentOS6 using Firefox ESR 10.0.4. The buttons all appear to do what they advertise and such. Some "bugs" I noticed:

  • There is some image "tearing" (not sure what the technical term should be). As the pendulum swings back-n-forth, sometimes bits and pieces of it are left behind. As the pendulum swings back over, the torn pieces slowly get cleaned up (usually needs a few "wipes" to get it right. Also interesting is that if I pan the image after the tearing, the torn pieces stay exactly where they were in figure coordinates. They do not move with the panning.
  • Zooming in works just fine, but zooming out by using the right mouse button triggers the Firefox context menu to appear. Note that the image still zooms out, you just have the context menu in the way. A wrinkle is that when the mouse cursor is over the context menu while holding down the button, the webagg backend doesn't receive mouse events, so it can't do anything until the button is let go, or the mouse moves away from the menu. For the zoom-to-rect feature, this doesn't really effect things, however, for any custom interactive graphs that queries for mouse_move (and maybe even mouse_over) events, this would be an issue. See additional note below.
  • When closing the browser tab, the python script still keeps running. I Ctrl-C'ed the script and it seems like it was stuck in Tornado's polling loop based on the traceback. Also, Firefox hung for about a minute after that.
  • Any plt.draw() or plt.figure() commands after a call to plt.ion() does not trigger a figure being "shown" (testcase: examples/mplot3d/wire3d_animation_demo.py)

Additional datapoint: mplot3d plots appear to work just fine, even the rotating and zooming. The context menu issue is very noticeable here because dragging the mouse while holding down the right mouse button does zooming for Axes3D plots. No zooming happens when the mouse passes over the context menu.

Michael Droettboom
Owner

@WeatherGod: To address your points:

I've been testing primarily on Chrome 23 and Firefox 16 on Linux. I just yesterday went back to Firefox 10 and have started working on some workarounds for that browser, but it's challenging.

  1. Firefox 10 doesn't support binary websocket messages (that was added in Firefox 11), meaning all of the messages have to conform to utf-8. Therefore this code contains an experimental hack to base64-encode the images and send them that way when Firefox 10 is detected. Unfortunately, it seems that there is a limit on the message size using this approach and so some frames get "skipped" -- and I've also seen it refuse to load the initial full image, which is usually the largest anyway. It's going to require more cleverness to get this working -- perhaps chunking the images over multiple messages. This is probably worth the effort as long as Firefox 10 is still the ESR, but at the end of the day it's not going to perform nearly as well as a browser that includes support for binary websockets.

  2. There are various hacks around the net to disable the context menu, but the current generation of browsers seem to make it impossible to disable it for security reasons. (It can be used to create a fake context menu that takes the user to bogus "View Page Source" content, for example). Therefore there is a workaround here to convert the middle mouse button to a right mouse button. That's not terribly ideal, and I think eventually I may separate "pan" and "zoom" into two toolbar button modes, each of which would only use the left mouse button. But as you point out, that doesn't address custom event handlers.

  3. I'm not sure that it's wrong that the server keeps running after the tab is closed. Multiple clients may be connected and want to disconnect and reconnect later. In the remote case, where the server and browser are on different machines, this would cause the server to quit if there was a network interruption. I think perhaps instead that a message should be displayed "Press Ctrl+C to stop server" and "Ctrl+C" should be caught so it doesn't provide a traceback or look like a failure in any way. I'm not sure why Firefox was hanging after that. I'll pay with that for a bit. Was that with Firefox 10?

  4. I haven't experimented with ion interaction -- I'll see what's up there.

Benjamin Root
Collaborator

Yes, all of the above was done in Firefox 10. The context menu issue is going to be tricky. I am not particularly keen on mapping to the middle button, but that may have to be what happens. We would need to advertise that fact. This may also cause conflicts with any existing code that utilized the middle mouse button already (I have to check to see if mplot3d was using it at all...).

The hanging issue with Firefox might have been spurious. I only noticed it happening very badly once, so it could have been unrelated.

Wolfgang Kerzendorf wkerzendorf referenced this pull request in ipython/ipython
Closed

Matplotlib html backend support #2579

Michael Droettboom mdboom referenced this pull request from a commit in mdboom/matplotlib
Michael Droettboom mdboom Address documentation and smaller issues raised by @pelson in #1426. ef72616
Phil Elson
Collaborator

@mdboom - I'm a big fan of getting new features of such value in quickly for early adopters to pick up and try out (and report bugs before the next release). I'd be willing to put some effort in this week to get this merged - if your keen, would you mind getting this into a state where you are happy with it, and I will do a thorough once over before merging (assuming everybody else is happy with that).

After that, I presume it is about getting similar functionality into the ipython notebook? Have you had an attempt, or are you hopeful that someone from the ipython dev team will meet us half way?

Cheers,

Michael Droettboom
Owner

I think this is not far from a shape where it should be merged. I spent a little time trying to support browsers that don't support binary WebSockets, but I'm not sure it's feasible -- it really complicates things. Firefox 17 LTS is now out, superceding the Firefox 10 LTS, so the issue that there's a Firefox LTS that doesn't support this is all but moot now. In any case we should add some browser detection code and warnings if the browser is likely too old.

Other than that, here's my TODO list:

  • Fix the HTML/CSS to look less clunky. I'm really not an expert in these areas, so hopefully someone can step up there. We can shamelessly steal the CSS from IPython to have a consistent look.

I think we can then merge this before tacking the IPython integration. I'm sure they're excited about this -- but I'm not sure about the best way to go about integrating this. Will need to bring some IPython experts on board.

Cameron Bates

@mdboom As far as I can tell in it's current state it would be rather difficult to integrate this in any direct way to an ipython notebook due to it's reliance on the tornado websockets since in ipython notebooks the kernel running the webserver is different than the notebook kernel, but @ellisonbg is the real expert on the notebook so he would be the one to ask.

This is already incredibly useful though as a way to have an interactive figure in a browser for a remote ipython notebook session. It might be good to change the default port though because currently it can't be used alongside an ipython notebook since they both try to use localhost:8888 and they get confused.

I think any direct integration of matplotlib into an ipython notebook will require the interactivity to be done using js callbacks to the local ipython kernel but I may be wrong.

Michael Droettboom mdboom merged commit 7ed17e3 into from
toddrjen toddrjen referenced this pull request from a commit
Commit has since been removed from the repository and is no longer available.
Michael Droettboom mdboom deleted the branch
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on Jan 4, 2013
  1. Michael Droettboom

    Initial work on WebAgg backend

    mdboom authored
  2. Michael Droettboom
  3. Michael Droettboom
  4. Michael Droettboom
  5. Michael Droettboom
  6. Michael Droettboom

    Handle non-binary image streams

    mdboom authored
  7. Michael Droettboom
  8. Michael Droettboom

    Remove some logging

    mdboom authored
  9. Michael Droettboom
  10. Michael Droettboom
  11. Michael Droettboom
  12. Michael Droettboom
  13. Cameron Bates Michael Droettboom
  14. Cameron Bates Michael Droettboom

    remove empty ipython notebook

    crbates authored mdboom committed
  15. Cameron Bates Michael Droettboom

    Add jquery ui hover hook for buttons

    crbates authored mdboom committed
  16. Cameron Bates Michael Droettboom

    Move status message to a new line

    crbates authored mdboom committed
  17. Michael Droettboom
  18. Michael Droettboom

    Indentation

    mdboom authored
Something went wrong with that request. Please try again.