Display systems

[davidanthoff]

@fredo-dedup Just saw your work in blink_display and wanted to share some ideas I had around that.

I started https://github.com/davidanthoff/BlinkDisplay.jl a while ago. The idea is that this is a package that pushes a display on the display stack and then can handle various MIME types. One could be the vega-lite MIME type (and we could also add support for the vega and plotly MIME types etc., essentially support for the same MIME types that jupyterlab handles). I think one nice aspect about that design would be that VegaLite.jl and BlinkDisplay.jl would not need to have a dependency in either direction, but they would still work together via the MIME type.

I also plan to add support for the vega-lite MIME type to the VS Code julia extension. And I think the Juno folks are also considering providing support for displaying figures via the base multimedia system.

I think if that all works out, it would be great for something like VegaLite.jl: essentially that package would only have to provide a couple of show methods, and then depending on the runtime environment (jupyterlab, nteract, Juno, VS Code, BlinkDisplay) things would just show properly.

I guess the corollary of that is that I think plotting packages like VegaLite.jl should not push things on the display stack. That will at the end of the day mess things up when run say in VS Code, IJulia etc., because it really is those kinds of packages that provide displays that should push them onto the stack, not plotting packages.

[fredo-dedup]

Thank you for the explanation. You are right that it makes sense to have the display backends packages (IJulia, BlinkDisplay) to push their Display struct on the displays stack and advertise which MIME types they can handle. It would be cumbersome to handle all possible displays in VegaLite.jl.
Are you suggesting that I should rather add this code to BlinkDisplay ? I can try to think about something and propose a PR there.

I have one question though : How is the user to chose its display if say, he/she is working in VS code but would like to have the plots show in a Blink window, or a browser window ? Perhaps there should be a way to let users rotate the displays on the stack so that their choice is on the top ?

Possibly related : Electron can probably be used as an engine to convert Vega-Lite plots to files (it was one the intentions I had by creating this branch) : a windowless instance of Electron could be launched , the plot rendered, and the image pulled from Electron as a png, svg or jpg. All of this intended as an alternative way to installing Node to render a plot to a file.
Do you think this could fit in the BlinkDisplay ?

[davidanthoff]

I made some progress with https://github.com/davidanthoff/ElectronDisplay.jl (and renamed it). It now supports the vega-lite MIME type out of the box, in the same way that IJulia supports it (if used via jupyterlab or nteract). I plan to add the same type of support to the VS Code extension for julia as well, and then we would just have to convince the Juno folks to also hook into that, and we would have pretty much all display systems covered, right?

I think you raise a really good point about how one can choose the right display... Maybe a solution would be to have another package that provides a convenient API for users to manipulate the display stack? DisplayManager.jl or something like that. It could have functions that provide the kind of rotating you describe etc., but it would not be tied to any specific display implementation. Individual display packages like ElectronDisplay.jl could even reexport its functions, potentially?

Oh, one more thing: I've created a fork of Blink.jl called Electron.jl and am using that. Blink.jl had a pretty involved architecture (lots of web server stuff, always triggered the windows firewall, some integration with Juno etc.), and really I wanted something simpler, with better test coverage etc.

I think one could use Electron.jl to do the offline rendering, but are there actually any problems with the node version? My sense is that it is working pretty reliably? I guess one could get rid of the Cairo/Rsvg dependency for PNG and JPG, but for PDF we would still need it, right? And PDF is probably pretty important... I'm still hoping that Cairo.jl and Rsvg.jl just start to use BinaryProvider.jl, and all of the current installation woes with them will magically go away ;)

[davidanthoff]

I think for now this is pretty much sorted. A new DisplayManager.jl package would be cool, but that would be pretty independent form VegaLite.jl. So I'm closing this issue for now.

[datnamer]

How about integration with https://github.com/JuliaGizmos/WebIO.jl?

That way you can use lots of display systems for free alongside compatibility with parts of the Julia web stack like mux, interact and maybe genie.jl.

@piever might have more to say on this :)

[davidanthoff]

I don’t think that would add much, to be honest. Maybe Mux integration, but does that even make sense?

For all the other front ends I think we have a pretty ideal story at this point, with almost no dependencies.

[piever]

As far as the plot is displayed as an svg, it actually seems to work out of the box (meaning it integrates smoothly in an Interact* GUI) for example:

using InteractBulma
@manipulate for show in true
    if show
        iris |> @vlplot(
            :point,
            x=:SepalLength,
            y=:SepalWidth,
            color=:Species,
            width=400,
            height=400
        )
    end
end

The main thing this would add is the possibility to add an interactive VegaLite plot to an Interact* GUI. Is interactivity already developed also on the Julia side? Is there a simple example I could copy paste to test things?

If interactivity doesn't work out of the box in Interact* GUIs, the solution would be to define a WebIO.render method for VegaLite plot objects (which would simply load the needed libraries this package already downloads and specify what is the relevant html to insert in the GUI). I don't think the WebIO dependency is necessary as overloading WebIO.render can be done in a @require block.

[davidanthoff]

Well, the idea really is that one specs interactive stuff as part of the plot itself: https://vega.github.io/vega-lite/docs/selection.html

[piever]

What I meant is that as long as the plot is embedded in the jupyter notebook as svg (which seems to me like the current default) the integration works smoothly, whereas for something that requires embedding some html with a link to some javascript library, WebIO needs to know the path to the library to embed things correctly. This PR for example is what it'd look like for PlotlyJS.

[davidanthoff]

The current default for Jupyter is that VegaLite.jl offers things as three MIME types: png, svg and vega-lite. And then it depends on the Jupyter client what it will use :) In Jupyter Lab and Nteract things will show nativly as vega-lite, because they ship that out of the box; everywhere else it will be svg or png.

I did look at other options a la embedding custom HTML, and back then I essentially concluded that those are more or less all hacks that all stop working in one or another scenario and at the end leave folks with a bad user experience, so I'm kind of of hesitant to go there... We are of course in a pretty luxurious situation here, because vega-lite just ships with many clients by default so we don't really need to sort out the deployment story.

[piever]

Custom HTML + javascript has two advantages:

• Embedding plots in websites and WebIO GUIs while keeping interactivity
• Saving plots locally as html and when reopened in the browser they're still interactive

Once the custom html version exists, it's probably straightforward to add support to WebIO, but whether that's worth the extra hassle / bugs is of course a different story...