JS Plugins and JSON Handlers #2518

ellisonbg · 2012-10-25T21:38:57Z

This add two new things to the IPython Notebook:

JavaScript plugins: third party JS+CSS code that plug into the Notebook.
JSON handlers: JS plugins can send JSON to the browser using publish_json and declare a JS function that will be the handler for that message.

See:

https://github.com/ipython/jsplugins

For details and an Online Python Tutor example.

Carreau · 2012-10-31T08:48:13Z

Looks fine to me.
Does this fall into the 'big JS refactor we shouldn't merge until testing is done' ?

https://github.com/ipython/jsplugins

Do you wan't all the plugin to be on one repository ?
I would prefere having ipython/pytutor-plugin, ipython/notebook-slideshow , ipython/.... as separate repository.

ellisonbg · 2012-10-31T20:19:25Z

I don't think we should stop Notebook development until testing is done.

On Wed, Oct 31, 2012 at 1:48 AM, Bussonnier Matthias <
notifications@github.com> wrote:

Looks fine to me.
Does this fall into the 'big JS refactor we shouldn't merge until testing
is done' ?

https://github.com/ipython/jsplugins
Do you wan't all the plugin to be on one repository ?
I would prefere having ipython/pytutor-plugin, ipython/notebook-slideshow,
ipython/.... as separate repository.

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/2518#issuecomment-9936954.

Brian E. Granger
Cal Poly State University, San Luis Obispo
bgranger@calpoly.edu and ellisonbg@gmail.com

jasongrout · 2012-11-13T19:24:28Z

It would be great if there were a way for the javascript to send messages back to IPython other than constructing a string and sending it in an execute request call. For example, it would be great if we could: (1) be able to register handlers on the python side for messages with a specific type (or some other metadata), and (2) send such custom messages from the javascript code.

ellisonbg · 2012-11-13T19:27:15Z

Yes, I have been thinking about this exact thing. I agree that right now
it is a bit of a pain to write code in strings - especially if you want to
pass JSON data back to the kernel.

On Tue, Nov 13, 2012 at 11:24 AM, Jason Grout notifications@github.comwrote:

It would be great if there were a way for the javascript to send messages
back to IPython other than constructing a string and sending it in an
execute request call. For example, it would be great if we could: (1) be
able to register handlers on the python side for messages with a specific
type (or some other metadata), and (2) send such custom messages from the
javascript code.

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/2518#issuecomment-10339196.

Brian E. Granger
Cal Poly State University, San Luis Obispo
bgranger@calpoly.edu and ellisonbg@gmail.com

jasongrout · 2012-11-13T19:37:54Z

Writing all code and data in strings is exactly what the IPython display messages eliminates (i.e., we have richer communication than just stdout strings). It makes sense to extend this capability also to messages back to python code.

RishiRamraj · 2012-11-14T17:48:07Z

At pycon we wrote a simple plugin. About the only issue we had was that the interpreter tended to eat all of the exceptions that were raised by the serialiser (the code that turned a python object into json). To fix this problem, we wrapped the serialisation code in an exception handler and sent the traceback up to the javascript, which alerted the exception; this feature should probably be built into the framework.

We actually spent most of our time fighting with javascript. I suspect javascript package management will become an issue as more complex javascript visualisers are written.

ellisonbg · 2012-11-14T18:08:50Z

Could you post a link to the exception handling code on this PR, so I can have a look.

RishiRamraj · 2012-11-14T19:30:33Z

We took down the handling code in the master branch once we had the serialiser functioning. I've rewritten a simple example on this branch of our project: https://github.com/RishiRamraj/seepymol/tree/feature/exceptions

Carreau · 2012-11-14T19:45:37Z

I see what you mean.
In python a @jsonhandler decorator that wrap the function in a try/catch and return the exception as JSON to the notebook. Actually we could say that on the wire, the json your decorated function will always be :

{ data : json retured by your function , 
  error : None, or serialized tracemback,
  metadata : maybe some metadata,
}

On js side we can make sure that only data is passed around.

if error is not none then IPython notebook itself could show the traceback.

RishiRamraj · 2012-11-14T20:02:17Z

Although I haven't given it too much thought, I think the structure could following something along the lines of the parameters defined for the exit function in PEP 343:

{
    self:
    type:
    value:
    traceback:
}

I'm not sure if a json handler would ever be interested in the error codes returned by the python serialiser.

ellisonbg · 2012-12-29T18:30:51Z

Here are the main things we need to work our for this PR:

Come up with a better way for users to install JS plugins that manually copying things over.
Review the format of the JSON objects that come back. Right now the content is mixed in at the same level as the handler declaration. Maybe a top level content attribute?

jasongrout · 2012-12-29T23:01:17Z

After working for a bit on implementing interactive javascript controls, I'd like the ability to register handlers for specific mime types, rather than an all-encompassing json type. That takes care of the namespace issues as well, since I can just invent my own mime type, like "application/sage-interact" or something. This addresses your second point---the handler is different than the content.

ellisonbg · 2012-12-30T04:58:14Z

Sounds a bit like the wild west. We would have to modify the entire
display architecture to allow arbitrary mime types to be displayed. I
don't think we want to go there. Part of the problem is that we can't
send arbitrary mime types back as we embed the content in JSON messages.
We have to add special handling/encoding for sending back non JSON
friendly mime type data such as images. But the more important question is
why can't you just use JSON? Isn't JSON perfect for this?

On Sat, Dec 29, 2012 at 3:01 PM, Jason Grout notifications@github.comwrote:

After working for a bit on implementing interactive javascript controls,
I'd like the ability to register handlers for specific mime types, rather
than an all-encompassing json type. That takes care of the namespace issues
as well, since I can just invent my own mime type, like
"application/sage-interact" or something.

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/2518#issuecomment-11758839.

Brian E. Granger
Cal Poly State University, San Luis Obispo
bgranger@calpoly.edu and ellisonbg@gmail.com

jasongrout · 2012-12-30T05:29:36Z

Well, then you're back to 'how do you specify the handler?' The mime type is a great way (in fact, the current way) to change between handlers.

I guess this is the same as the custom message discussion earlier, but one level of wrapping down from what we were discussing earlier. Do we want to wrap custom messages (your solution of using a json mime type) or give users a way to change the top-level msg type (my solution of allowing custom mime types).

You're right that the current system really isn't using mime types correctly, since there isn't a way to ship purely binary data over json (e.g., the image/png mime type really isn't binary image data...). So your argument about the data not actually being the right mime type actually extends to the current system (at least if you wanted to be strictly pure). What happens now is that you indicate the intent/purpose of the data with the mime type, rather than the actual format, and it's up to the client to understand how you encoded the mime type to fit JSON.

I agree that specifying the handler at the same level as the content is not very elegant. The handler should be specified above the content level. Another natural place to specify the handler would be the metadata for the entire message. What do you think about using the top-level metadata dict for specifying a handler? Or should that metadata dict be strictly preserved for the user?

ellisonbg · 2012-12-31T04:15:28Z

I guess this is the same as the custom message discussion earlier, but one
level of wrapping down from what we were discussing earlier. Do we want to
wrap custom messages (your solution of using a json mime type) or give
users a way to change the top-level msg type (my solution of allowing
custom mime types).

In my mind JSON is just JSON and doesn't warrant a new message type. I
think we should just make the message look like:

{'handler': handler_name, 'content': {actual data goes here}}

You're right that the current system really isn't using mime types
correctly, since there isn't a way to ship purely binary data over json
(e.g., the image/png mime type really isn't binary image data...). So your
argument about the data not actually being the right mime type actually
extends to the current system (at least if you wanted to be strictly pure).
What happens now is that you indicate the intent/purpose of the data with
the mime type, rather than the actual format, and it's up to the client to
understand how you encoded the mime type to fit JSON.

I agree that specifying the handler at the same level as the content is
not very elegant. The handler should be specified above the content level.
Another natural place to specify the handler would be the metadata for the
entire message. What do you think about using the top-level metadata dict
for specifying a handler? Or should that metadata dict be strictly
preserved for the user?

I didn't go the metadata route because currently don't have a a clean way
to send metadata with JSON. Even if we did, I am not sure if this type of
thing falls into the metadata category or not.

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/2518#issuecomment-11761457.

Brian E. Granger
Cal Poly State University, San Luis Obispo
bgranger@calpoly.edu and ellisonbg@gmail.com

jasongrout · 2012-12-31T15:57:56Z

Are you proposing a message format of: {parent: {...}, header: {...}, metadata: {...}, content: {handler: {...}, content: {mimetypes}}, or {parent: {...}, header: {...}, metadata: {...}, handler: {...}, content: {mimetypes...}? In other words, are you proposing adding a new top-level attribute of handler, or are you proposing wrapping the mimetypes one level deeper in a display_data message?

Really, the mimetypes really are about selecting a handler on the client side, and the handler knows how to interpret the value of the key. You could also do a special mimetype of application/ipython-<handler>, where handler is filled in.

If you're proposing a handler attribute at the top level, then would it also be possible to specify a handler for messages other than display_data messages? Would it be possible to specify a handler for display_data messages that only had, say, a text/plain mimetype?

ellisonbg · 2013-01-07T20:52:41Z

I am thinking that the main content dict would have two subfields: handler
and content. I like that because it makes it very easy for people to
specify the handler by building the right JSON object in their application
code - IOW, they don't have to call a special IPython API to pass it to the
metadata.

I also don't want to get into allowing custom handlers for arbitrary data
types - just the JSON ones. The other types have handlers that are
complete determined by their type. This make sure that there is a uniform
handling of these types that can be used to build consistent notebook
documents = png's are always handled in this particular way.

On Mon, Dec 31, 2012 at 7:58 AM, Jason Grout notifications@github.comwrote:

you're proposing a message format of: {parent: {...}, header: {...},
metadata: {...}, content: {handler: {...}, content: {mimetypes}}, or {parent:
{...}, header: {...}, metadata: {...}, handler: {...}, content:
{mimetypes...}? In other words, are you proposing adding a new top-level
attribute of handler, or are you proposing wrapping the mimetypes one
level deeper in a display_data message?

Really, the mimetypes really are about selecting a handler on the client
side, and the handler knows how to interpret the value of the key. You
could also do a special mimetype of application/ipython-, where
handler is filled in.

If you're proposing a handler attribute at the top level, then would it
also be possible to specify a handler for messages other than display_data
messages? Would it be possible to specify a handler for display_data
messages that only had, say, a text/plain mimetype?

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/2518#issuecomment-11779352.

Brian E. Granger
Cal Poly State University, San Luis Obispo
bgranger@calpoly.edu and ellisonbg@gmail.com

jasongrout · 2013-01-07T21:16:01Z

A couple of clarifying comments:

The main content dict currently has "source", "data", and "metadata" keys (but no "content" key), according to the spec. Are you saying it should have a fourth key?
Will the handler key be ignored if a JSON datatype is not passed? What if JSON and png and text/plain content types are included? Is the handler passed the entire message, or just the json mimetype content? Is the png handler called as well?

The difference between our two approaches seems to be just that I want to specify the intent of the content using the mime type, whereas you want to specify the intent using the mime type and a handler key. Is that a fair characterization?

I think my approach plugs into the existing system very well and is simpler (one way of registering handlers for the message data). An implementation detail can say that the standard mimetypes are not available to be overridden. I think making the handler request disconnected from the data (e.g., having the handler specified separately from the content) is not as elegant as making the handler an integral part of the data (e.g., having the handler specified as a key with the content as the data).

Also, if the handler is specified using the mimetype system, you can set up multiple handlers for the same message. What if, for example, I have a python library that can send data back and use one of three different custom handlers. Having the handler being specified by a key in the data dictionary means I can consolidate the display data in one message, and the client can decide which (or all) of the handlers called. Under your proposal, I'd have to send several different display messages, and all would be acted on.

ellisonbg · 2013-01-07T22:10:37Z

No, I mean that the "data" field on content would have the handler
attribute. The problem is that the metadata, source, etc are not exposed
through any API to the code sending the message. That way the handler
attribute only makes sense when data is, itself JSON data. We could put it
up one level, but it would require reworking a good part of our message
sending API in possibly backwards incompatible manner. The problem is that
right now the various message attributes are not really exposed very well.
If we were doing this from scratch, the handler would definitely go into
the metadata dict.

On Mon, Jan 7, 2013 at 1:16 PM, Jason Grout notifications@github.comwrote:

A couple of clarifying comments:

The main content dict currently has "source", "data", and "metadata"
keys, according to the spec. Are you saying it should have a fourth key?
2.

Will the handler key be ignored if a JSON datatype is not passed? What
if JSON and png and text/plain content types are included? Is the handler
passed the entire message, or just the json mimetype content? Is the png
handler called as well?

The difference between our two approaches seems to be just that I want to
specify the intent of the content using the mime type, whereas you want to
specify the intent using the mime type and a handler key. Is that a fair
characterization?

I think my approach plugs into the existing system very well and is
simpler (one way of registering handlers for the message data). An
implementation detail can say that the standard mimetypes are not available
to be overridden. I think making the handler request disconnected from the
data (e.g., having the handler specified separately from the content) is
not as elegant as making the handler an integral part of the data (e.g.,
having the handler specified as a key with the content as the data).

Also, if the handler is specified using the mimetype system, you can set
up multiple handlers for the same message. What if, for example, I have a
python library that can send data back and use one of three different
custom handlers. Having the handler being specified by a key in the data
dictionary means I can consolidate the display data in one message, and the
client can decide which (or all) of the handlers called. Under your
proposal, I'd have to send several different display messages, and all
would be acted on.

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/2518#issuecomment-11972082.

Brian E. Granger
Cal Poly State University, San Luis Obispo
bgranger@calpoly.edu and ellisonbg@gmail.com

jasongrout · 2013-01-07T22:18:21Z

I'm still a little confused. Can you post an example message that specifies a handler? In particular, are other mimetypes allowed in the data dictionary?

ellisonbg · 2013-01-08T02:58:35Z

I'm on my phone so it is a little bit difficult to type a Json structure. The display data message type has a data attribute this data attribute would contain the Json data for the message being sent to the browser. I propose creating a handler sub attribute within the data structure that would contain the handler for that data. The actual Json data would sit alongside the handler attribute in the data sub structure. I can describe this more when I get home if you want. This is also how it implemented it in my JS plug-ins branch. Who Ray for voice recognition!

Sent from my iPhone

On Jan 7, 2013, at 2:18 PM, Jason Grout notifications@github.com wrote:

I'm still a little confused. Can you post an example message that specifies a handler?

—
Reply to this email directly or view it on GitHub.

jasongrout · 2013-01-08T03:08:13Z

So like this?

{header: {...}, 
parent: {...}, 
metadata: {...}, 
content: {data: {'text/json': jsondata, 'handler': handler_name, 'text/plain': 'json plugin'}}}

ellisonbg · 2013-01-08T06:26:39Z

No I am sorry I forgot one detail. Here is how it is now:

{header: {...},
parent: {...},
metadata: {...},
content: {data: {'text/json': {handler: 'myhandler', content: {rest of JSON data}}}}}

IOW, the handler attribute is part of the actual JSON payload.

ellisonbg · 2013-01-18T00:47:47Z

I am thinking about closing this PR. There are a number of larger design issues that we really need to work out first and those discussions are going beyond the scope of this PR. How do people feel about this (I would open an issue to track the larger issues).

jasongrout · 2013-01-18T01:52:20Z

+1 to opening up the broader discussion

ellisonbg · 2013-01-18T03:53:48Z

I am closing this issue as there are larger design questions that have to be worked out. Follow that discussion at #2802.

ellisonbg added 4 commits October 24, 2012 16:57

First go at implementing JSON output handlers.

e66fd8c

Adding JS Plugins to enable people to include custom JS easily.

3268f25

Adding comment about JS plugins.

8764647

Use os.walk to find all .js and .css files anywhere in jsplugins.

08f7f0f

Handling JSON first and fixing when it is rendered for pyout.

7a2694c

Carreau mentioned this pull request Nov 17, 2012

Automatically include Python environment info in notebook metadata #2597

Open

eloraburns mentioned this pull request Dec 18, 2012

Working Skip support eloraburns/ipython_nose#3

Merged

ellisonbg mentioned this pull request Jan 18, 2013

Add JSON handlers and JS plugins to the Notebook #2802

Closed

3 tasks

ellisonbg closed this Jan 18, 2013

bollwyvl mentioned this pull request Apr 14, 2013

Integrate the Hem Assets toolchain into the IPython notebook #3171

Closed

ellisonbg deleted the jsonhandlers branch November 20, 2013 21:52

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

JS Plugins and JSON Handlers #2518

JS Plugins and JSON Handlers #2518

ellisonbg commented Oct 25, 2012

Carreau commented Oct 31, 2012

ellisonbg commented Oct 31, 2012

jasongrout commented Nov 13, 2012

ellisonbg commented Nov 13, 2012

jasongrout commented Nov 13, 2012

RishiRamraj commented Nov 14, 2012

ellisonbg commented Nov 14, 2012

RishiRamraj commented Nov 14, 2012

Carreau commented Nov 14, 2012

RishiRamraj commented Nov 14, 2012

ellisonbg commented Dec 29, 2012

jasongrout commented Dec 29, 2012

ellisonbg commented Dec 30, 2012

jasongrout commented Dec 30, 2012

ellisonbg commented Dec 31, 2012

jasongrout commented Dec 31, 2012

ellisonbg commented Jan 7, 2013

jasongrout commented Jan 7, 2013

ellisonbg commented Jan 7, 2013

jasongrout commented Jan 7, 2013

ellisonbg commented Jan 8, 2013

jasongrout commented Jan 8, 2013

ellisonbg commented Jan 8, 2013

ellisonbg commented Jan 18, 2013

jasongrout commented Jan 18, 2013

ellisonbg commented Jan 18, 2013

JS Plugins and JSON Handlers #2518

JS Plugins and JSON Handlers #2518

Conversation

ellisonbg commented Oct 25, 2012

Carreau commented Oct 31, 2012

ellisonbg commented Oct 31, 2012

jasongrout commented Nov 13, 2012

ellisonbg commented Nov 13, 2012

jasongrout commented Nov 13, 2012

RishiRamraj commented Nov 14, 2012

ellisonbg commented Nov 14, 2012

RishiRamraj commented Nov 14, 2012

Carreau commented Nov 14, 2012

RishiRamraj commented Nov 14, 2012

ellisonbg commented Dec 29, 2012

jasongrout commented Dec 29, 2012

ellisonbg commented Dec 30, 2012

jasongrout commented Dec 30, 2012

ellisonbg commented Dec 31, 2012

jasongrout commented Dec 31, 2012

ellisonbg commented Jan 7, 2013

jasongrout commented Jan 7, 2013

ellisonbg commented Jan 7, 2013

jasongrout commented Jan 7, 2013

ellisonbg commented Jan 8, 2013

jasongrout commented Jan 8, 2013

ellisonbg commented Jan 8, 2013

ellisonbg commented Jan 18, 2013

jasongrout commented Jan 18, 2013

ellisonbg commented Jan 18, 2013