Dashboarding

[jasongrout]

To keep this archived somewhere, here are the notes from our discussion on dashboarding at the ipywidgets workshop in Paris in Jan 2018 (see OpenDreamKit/OpenDreamKit#246, and https://www.lri.fr/etherpad/p/jupyter-widgets for the notes).

We can get an overview of current solutions to displaying the widgets on a web page, as sort of a web app. For example, Pascal's ipywidget-server, https://github.com/oschuett/appmode, or #1862

Pascal's solution: a separate server that is instantiated with the code to be run, a specific kernel https://github.com/pbugnion/ipywidgets_server

https://github.com/oschuett/appmode - strip out the classic notebook UI

#1862 (not finished) - read a notebook, execute all cells from the browser, display specific cell outputs

IBM Jupyter Dashboard server - easy drag-and-drop layout of cell outputs in a grid, one-click publish, secure execution of the notebook by not accepting code execution from the browser
https://github.com/jupyter-attic/dashboards_server and https://github.com/jupyter/dashboards and https://github.com/jupyter-attic/dashboards_bundlers

Do we reuse kernels (e.g., you give a link to someone with a widget from a specific running kernel), or start a new kernel for every dashboard view

• Different kinds of uses cases: dashboards as "isolated" kernel execution (each dashboard has own kernel) vs dashboard as "broadcast" enviornment - by interacting with one I am affecting all others.

Can we expose a layout widget that lets you drag and drop to layout things, and then save the layout.

Integration with kernel gateway / remote kernel manager?

It's not obvious we need a live separate kernel. For some usecases, you may want to execute the code in-process (e.g., flask, django).

Should the layout depend on widgets and the kernel? Or should the layout be independent of widgets/kernel.

• Making two layout systems can be confusing (a frontend UI layout, and a widget layout)
• We could make a layout widget that lets you drag and drop children to lay out them. Then save the state of the widget.

Do we force the authoring environment to be a notebook? An authoring environment like shiny?

Perhaps we just have some very specific predefined layouts that can just be populated with children.

If we were able to publish a binder-ready repo for a dashboard, that could take care of a deployment usecase.

Needs:
- Subject-matter experts (know python, not js) easily expose their code/explorations as a webapp to stakeholders with no programming skills, so they don't want any notion of a notebook, code, etc. Code might be running internal software stacks, etc., so we don't have to distribute software, just expose a GUI to use the tools.
- Students exploring learning materials (similar to above usecase)
- Should be usable from different language kernels, such as R
- Layout should be doable programmatically
- E-posters at an academic conference (three-column layout)
- Include markdown/text/html, possibly with variable substitutions
- Security - not being able to execute arbitrary code from the browser - needs to be an option
- Programmatic layouts should use standard layout protocols, e.g., flexbox, grid, etc.
- Other styling, such as colors, fonts, etc.

Here is another issue talking about dashboarding: jupyterlab/jupyterlab#1640

[jasongrout]

Actually, reopening this for further discussion too.

[maartenbreddels]

I hould also mention https://github.com/maartenbreddels/flask-ipywidgets which is a proof of concept of inprocess kernel just doing Comms for flask, which should be able to port to Django and tornado. I think this, ipywidgets-server and appmode could live in 1 repo since they will share a lot of code.

[ixxie]

We have been working on an ipywidgets dashboard as a way to quickly get an internal GUI tool going; our setup (outlined here ) is a Jupyterhub server with Jupyterlab with a shared group owned directory symlinked into users' notebooks directories (I believe this is quite a common approach to sharing notebooks).

So far out major pain-points are:

1. Having to restart the kernel and run all cells to render the dashboard; this has to be done on every page load (at least I haven't found a way around it).
2. It seems that the notebook saves some kind of state to disk, because concurrent use of the notebook by multiple users triggers the 'revert file' dialogue box.
3. We cannot seem to find a reliable and elegant way to share notebook URLs; this seems to work well for the classic tree links but with lab the interface saves some kind of viewport state which gets loaded.

It seems to me - with my addmitedly limited knowledge of Jupyter ecosystem internals - that a sensible and relatively easy to implement solution would be to create an .ipyapp format which is mostly just like a normal .ipynb notebook but:

1. Has only one input cell which is always hidden and run when the page is loaded;*
2. Saves no state to disk, other than whatever the user explicitly tells the dashboard to save;
3. Runs in 'appmode' (i.e. strips out the notebook UI) and has an affiliated reliable URL that launches the app, like the appmode extension;**
4. Gets added to a new 'apps' section of the launcher (with an optional user specified icon).

Does that make any sense?

---

* I guess this means the notebook would be 'trusted'?
** appmode doesn't seem to run correctly on our stack; we get 404 errors.

[ixxie]

I would be happy to try and contribute that feature with some guidance; I have some experience with Python but not so much Java/Typescript.

[oschuett]

Hi @ixxie, it seems that all your pain-points would be addressed by adding support for JupyterLab to appmode. This is something I've been putting off because the API was still in flux. I'll now put it back on my TODO list. However, it'll probably take me a few weeks to carve out the time. So, feel free to take a first shot at it yourself.

[ixxie]

@oschuett I wouldn't know where to begin, but since this is a nice way to get introduced to Jupyter/Lab extension writing, perhaps you can sketch out a rough implementation plan and I could maybe give it a shot over the next weeks?

But I wonder whether appmode - one updated to support JupyterLab - could actually become a core feature of JupyterLab? It seems like something that would make sense to integrate into the core offering, and would make it easy to then specify a .ipyapp format spec.

[ixxie]

On another note, it seems our team's developers are now leaning towards adopting Dash as our dashboarding framework since it seems at least to allow us more flexibility (it binds the React framework into Python); it seems Dash works within Jupyter notebooks as well.

This means that once we get appmode going, we can deploy Dash apps. What if it is equally easy to deploy Flask or Django apps via JupyterLab? With some kind of deployment framework (Docker/Kubernetes integration) we would have something ready to deploy Python apps to production.

I would love to contribute in this area, but it would take me some time as I have limited knowledge of the necessary prerequisites. Maybe starting with an attempt to port appmode to JupyterLab would be the place to start :)

[BexDunn]

interested to hear where this goes

[rserbitar]

Any news?

[oschuett]

Sorry, I did not manage to get back to this project - it keeps getting preempted by other open source responsibilities of mine... So, don't hold your breath for me.

In any case, Appmode for JupyterLab will be a completely independent implementation. So, please go ahead and give it a shot if you like.

Of course, integrating Appmode directly into the JupyterLab core would open up a lot of exciting possibilities.

[ixxie]

I also didn't get around to this since our team has decided to opt out of a Jupyter-based UI and instead work directly with Javascript. I think there is a lot of potential for Jupyter widget based dashboards and Python based data science UI design, but some significant work is still needed before its viable in production.

[jasongrout]

@ixxie - interesting. Does that mean they also opted out of Dash, and are just doing a pure JS frontend? Are they doing stuff in a python kernel? I'm curious how they are communicating between js and python if they are.

[ixxie]

@jasongrout we tried out Dash but what we really interested in is stuff like phosphorjs datagrid, D3, deck.gl or sigmajs and the rest if the great visualization ecosystem. We also want finegrained control of the interface.

Regarding communication, the idea is still vague and untested, but the plan is to have a REST API and/or message queue to the python container. Our application does not require particularly rapid communication, and we can do client side Javascript for visualization and UI.

All this makes me wonder if the ultimate solution is to use something like Transcript but then a proper typing system like that of Typescript of Purescript is nice to have. Unfortunately we can't afford to spend our innovation points on an unfamiliar frontend technology.

Note people - myself included - were generally unhappy with having the Jupyter ecosystem as a dependency to the frontend, in part because I had some trouble maintaining my Jupyterhub+Lab server*.

One direction I would love to see Jupyterlab going is in integrating with frontend technology: live frontend development in the lab. Perhaps then UIs could be launched as apps in lab which talk through a generic protocol with the kernels? I don't know enough about the internals to answer that. If a headless kernel deployment is elaborated the apps could be deployed without the lab UI as well.

* I am hoping that switching to a dockerized deployment will reduce this pain. We will be adapting the existing template to our needs. One day maybe a Kube deployment too.

[jasongrout]

a headless kernel deployment

I think that is going to be a key to a lot of this conversation - an easy way to provision and spin up kernels. I have hopes for a mybinder-like environment for doing something like that.

[mathematicalmichael]

Hi, I know I'm late to the discussion but I've spent a really fair amount of time creeping on the issues for a number of related projects around this theme of dashboarding. I've read the roadmap and poked around the notes from the meetings.

Here's a little background on me and my use-cases:

• Doctoral student in Applied Mathematics, interested in communicating my results via online publishing

• Heavy python user, huge lover of Notebook/Lab/Hub, primary cheerleader/proponent for their adoption in my math department at CU Denver.

• Just successfully deployed my first set of hubs, live and secured on a domain I bought, both with and without docker, but I settled on Hub in Docker + Dockerspawner with auto-removed containers but persistent volumes. This took about an entire dedicated week to learn. Working to get it set up on private-network math server next and deploy (without kubernetes) for classes next semester.

• Struggled a lot learning server communication, still not entirely clear to me how to embed interactive content into static webpages and have it send the required commands to a kernel i have running somewhere (much the way nbinteract does with mybinder.org).

• What I would love to be able to do is communicate the utility of math by publishing simple "dashboard" embeds within "blog"-like posts.

• Personally, I liked the functionality of nbinteract (when it worked), where the notebook could easily be embedded in my static webpage and (albeit super slow) binder would handle the computations. It would be great if it were more predictable/reproducible. This is probably the thing that most people in my department could benefit from, since they can host the output on their webpages pretty easily by simply moving a single html file.

• Documentation is scarce on non-kubernetes deployments of binder. Would be great to just have a docker image that spun up a container with the bare minimum of necessary kernel environments alongside a server listening for requests. Starting to sound like a connection between Flask and ipywidgets that abstracts away most complexity. (Exporting could lead to a Dockerfile with the instructions for the image?)

• Dash is very impressive. But requires learning a new framework, and won't really be reproducible by others in my department (it was a struggle getting people on-board with widgets to be honest). I just finally figured out that I can deploy these apps on my server without having to pay the company (this whole business model vs open-source distinction is really unclear on their website. For someone without devops experience). However, I suppose with an iframe, I could embed these into static pages. For basic blog-posts, this could work. But it's not as ideal since I develop a lot of my work with widgets already.

• Building "interfaces"/GUIs with widgets is fantastic and almost surely the easiest way to share content with non-programmer users, perhaps sharing them via a jupyterhub or binderhub link. If one has patience, this works well. But ideally, there would be a way (like some of the previous dashboard attempts) to divorce the widgets from the rest of the notebook and embed them within a website.

Solutions like the one @ixxie is discussing are actually kind of what I've come around to... have a friend help write some front-end interfaces with D3 and then use a Flask server to handle computations. But man... someone shouldn't have to get that deep. I think it'd be fantastic to be able to create a dashboard for a presentation, perhaps embed it into a presentation with RISE, or even more broadly, allow people like myself to more readily develop "apps" to use within a company, or as standalone web-deployments. I think @jasongrout's suggestion for a mybinder-like environment is right. The complexity around setting something up is a major barrier to entry, and mybinder.org has to re-build images, which slows it down. The time I took to learn docker and jupyterhub is something most people in my department would not take on themselves. But the desire to share results online more easily is strong.

• It appears to me that this is what Thebe is doing?

That's some of my input. Hope it was a helpful insight. I really love what the Jupyter Project is doing and would love to get involved in some way, but I am unfortunately not a web developer (despite having learned a lot of it at this point). I feel I could help improve documentation a bit as a contribution from a user's standpoint? Let me know if I can be helpful some way.

[jasongrout]

Thanks for your thoughts, especially how comprehensive they are!

In addition to thebe, you might be interested in looking at the new Voila project that @SylvainCorlay and company are working on: https://github.com/QuantStack/voila

[mathematicalmichael]

No problem, and thanks for that link! I'll most definitely be checking it out.
Just out of curiosity, is it a concern that thebe hasn't been touched in a few years, or is the framework on which it is based simple/extensible enough that it still works even with newer versions of jupyter? Voila seems promising since it's new/active. Will be attempting to deploy.

[maartenbreddels]

For a reboot of thebe, you can take a look at https://github.com/minrk/thebelab
I think the approach of voila (and thebelab) to be light weight will make them future proof. Voila is really trying to combine the several jupyter project to implement dashboarding with minimal amount of code (we try to push improvements in the depending repos). Expect more development on voila in 2019, feedback is welcome (issues, new use cases etc), and I hope to do some work on https://github.com/maartenbreddels/ipymaterialui for a rich widget support as well.

[mathematicalmichael]

@maartenbreddels thanks for that! Yeah, I've just been playing around with voila and besides having to reformat widgets to avoid interact(ive), it seems to work well. But from what I can tell it won't quite let me embed into a static website, would it?

Thebelab definitely seems almost identical to what I need (though it would be nice to hide the code cells), and I like @minrk's "minimal" examples because they clarify what needs to be included to work in my website.

It's almost like a need a hybrid of the two. Metadata like from Thebelab telling the page where an active notebook environment with live kernel can be found, but then the content looks like the html page voila is serving. Ideally I'd be able to parse the html file, strip it of all but the necessary headers, and paste the body into a page on my website. (I use a static generator, so can embed the body/header html with a shortcode).

[jbednar]

@mathematicalmichael , note that Panel supports export to a static website (for feasible quantities of data), or running a separate web server (with Python backing), or embedding into a static website. It's not ipywidgets-based, but is designed to work well in a Jupyter Notebook as well as in these contexts, trying not to tie anything too deeply to the notebook context. Something to consider as an alternative to both ipywidgets and Dash, covering what both of those two libraries are used for but from a single API.

[dmadeka]

@jasongrout @SylvainCorlay Whats the status of this for my curiosity? Is there a currently accepted solution to exporting a dashboard?

[jbednar]

2019 and 2020 have been very busy years for Python dashboarding. There are now lots of options for dashboards; see e.g. https://pyviz.org/dashboarding/ (but I notice that it's already out of date, as https://www.streamlit.io needs to be added to it). My group is also about to announce our work making ipywidgets integrate fully with Panel and Bokeh, which gives you even more options. But the net result is there not being a single accepted solution, which I admit can be confusing for users. We've attempted to explain the pros and cons of the various solutions at https://panel.holoviz.org/Comparisons.html , but that page is both our own personal viewpoint (though we try to be as objective as we can!) and also perpetually out of date (as each dashboarding library races to fill in features that appear in other libraries!).

[SylvainCorlay]

@jasongrout @SylvainCorlay Whats the status of this for my curiosity? Is there a currently accepted solution to exporting a dashboard?

@dmadeka please check out Voilà, which is now an official Jupyter subproject.

[dmadeka]

Thanks! Voila was exactly what i was looking for!