Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

MIME based adaptor function #17

Open
ellisonbg opened this issue Sep 29, 2017 · 17 comments
Open

MIME based adaptor function #17

ellisonbg opened this issue Sep 29, 2017 · 17 comments

Comments

@ellisonbg
Copy link
Contributor

Python objects can have special methods (_repr_mimebundle_, _repr_html, _repr_png_, etc.) that could be called to grab content that could be easily adapted to their VDOM based components. This would be a nice way of doing general purpose adaptation between other objects and the VDOM API. We could have a single function vdom_adapt that handles all this:

def vdom_adapt(o):
    if hasattr(o, '_repr_png'):
        velement = img(src=o._repr_png_()) # ok, put in the data uri base64 encoded
    elif ...
    return velement

Then you could do things like:

div(IPython.display.Image('coolthing.png'), my_matplotlib_figure, ...)

We might need a way of selecting which MIME type to use if an object had multiple.
@rgbkrk
Copy link
Member

rgbkrk commented Sep 29, 2017 •
edited
Loading

That is such a great idea! All the image reprs seem like easy choices.

_repr_html_ may be a dangerous one to handle for two reasons:

  1. dangerouslySetInnerHTML does not actually run all scripts within since it's using innerHTML, which means it won't behave the same as how the current frontends handle html mimetypes (notebook uses $.html, nteract creates a document fragment)
  2. I was really hoping we'd try to keep this as sanitary as possible to continue mitigating security concerns in the long term

That being said, it sure would be nice to be able to group up pandas dataframe html into these outputs. The main use case that I'm 💯 % likely to use this for is generating stylized reports. My current thinking was that I would iterate over df.to_dict(orient='row'), using the table schema to assist in choosing formatting.

We might need a way of selecting which MIME type to use if an object had multiple.

We could do our classic display order handling here, matching that of the classic notebook as a sane default.

@ellisonbg
Copy link
Contributor Author

ellisonbg commented Oct 3, 2017 via email

@rgbkrk
Copy link
Member

rgbkrk commented Oct 3, 2017

I feel like if we use the IPython formatter API, we can use text/plain. Technically we could turn application/javascript into script(code), though I'd much rather stick script on the blocklist for the frontend handling of vdom.

@ellisonbg
Copy link
Contributor Author

ellisonbg commented Oct 3, 2017 via email

@rgbkrk
Copy link
Member

rgbkrk commented Oct 7, 2017

Ok Brian, I tried something a little crazy...

screen shot 2017-10-06 at 5 29 22 pm

Would it really be all that bad for us to parse some HTML or SVG 😉?

@ellisonbg
Copy link
Contributor Author

Not too crazy - and probably going to be hard to stop folks from wanting this. Would allow us to handle the SVG and HTML MIME types.

@mpacer
Copy link
Member

mpacer commented Oct 7, 2017 •
edited
Loading

This approach could work with audio and video as easily as it does with images.

I think where it gets murky is when you have to do a lot of interpretation of the "type" of thing that it is and a lot of recursion once you know that that type is.

Images, audio and video are all direct data forms that have a conventional way of being displayed that requires relatively little interpretation and (at least as far as I know) there's no way to nest images, audio or video (and so no recursion is needed).

General html (on the other hand) requires interpretation (to determine the first element type, assuming it's actually a fragment and not a full-blown page) and then recursion, since it by definition is able to hold other objects.

SVGs initially require recursion, and the elements below it are not guaranteed to be of the svg type, so even though the top-level svg tag doesn't require interpretation, all of its elements would.

I feel like this risks becoming heavy weight in practice if you ever wanted to update deeply nested objects in this way.

The caveat to that heavy-weight-ness would be if we can intelligently figure out incremental updates to pass back up, but that seems like a really hard problem in the general case that we probably don't want to have to invent/reinvent a solution to.

@rgbkrk
Copy link
Member

rgbkrk commented Oct 7, 2017

@mpacer is spot on in describing the potential complexity. Given my current stretch of maintained projects (which I admit I allow to grow), I'd prefer to keep this one simple. M is thinking ahead about if/when we handle updating nested objects (a virtual dom diff).

One way I'd probably think about this: React never takes arbitrary Objects -- you're expected to create a function that returns React elements or a class with a render function. That would keep it simple for us, which could mean we look for only _repr_vdom_ as if that was our render function. Note that we're probably going to end up going through multiple revisions of the vdom mimetype, so I don't know what that's going to look like...

@rgbkrk
Copy link
Member

rgbkrk commented Oct 7, 2017 •
edited
Loading

How about, for now, we create another repo/package, which we can toss any time we want, called vdomify. It would use the display_formatter to compute the mime bundle:

ip = get_ipython()
ip.display_formatter.format(obj)

Then we'd have our own "display order".

@ellisonbg
Copy link
Contributor Author

ellisonbg commented Oct 7, 2017 via email

@rgbkrk
Copy link
Member

rgbkrk commented Oct 8, 2017

Ok I made an initial stab at this in vdomify (which is also pip-installable -- run pip install vdomify).

I'm cheating with the HTML mimetype for now, I'll add image types shortly... The interface underneath pretty much matches https://github.com/nteract/nteract/tree/master/packages/transforms#standard-jupyter-transforms

screen shot 2017-10-08 at 1 36 27 pm

Which... I suppose now that I see a clean path for overriding our opinionated defaults, I'm a little more ok with it.
¯\_(ツ)_/¯

@ellisonbg
Copy link
Contributor Author

ellisonbg commented Oct 8, 2017 via email

@rgbkrk
Copy link
Member

rgbkrk commented Oct 8, 2017 •
edited
Loading

Yeah I made transforms for text/plain, text/html, and application/vdom.v1+json (which is a no-op). Open to the others.

It's kind of funny though, they mostly mirror the mimetype handling on the frontends (even the image/* ones will once implemented). Makes me kind of wish for either custom elements (web components) or a way to use a "namespace" of react components (note: we've been down the path before with the namespacing and was not pleased with the security ramifications if dynamically loaded).

@ellisonbg
Copy link
Contributor Author

ellisonbg commented Oct 8, 2017 via email

@rgbkrk
Copy link
Member

rgbkrk commented Oct 8, 2017 •
edited
Loading

I know there will be pressure from ourselves and others to allow custom components. I think it is reasonable to hold a hard line now and now go that way. Is easy to reconsider later, but once they are allowed,...you know how that goes...

Yeah, I'd like to see the simple basics get integrated first. We evangelize how to work with vdom, show people how to create their own components, and how to do display updates with them.

@willingc
Copy link
Member

Interesting discussion on gitter ipywidgets re: repr_html

From Matt Rocklin:

OK, so I have a semi-related question. Sometimes the data that I want to render has a _repr_html_ method defined. Sometimes it doesn't. In both cases I would like to render this thing with an HTML widget and get the same result as if the user had just rendered the value normally in the notebook.

@rgbkrk
Copy link
Member

rgbkrk commented Oct 11, 2017

Something like this:

screen shot 2017-10-10 at 10 17 16 pm

Or were you thinking of a way to do it with display updates?

screen shot 2017-10-10 at 10 22 17 pm

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@ellisonbg @rgbkrk @mpacer @willingc