Join GitHub today
GitHub is home to over 36 million developers working together to host and review code, manage projects, and build software together.Sign up
A couple of questions to start the discussion:
Working on multiple projects that are basically glorified dashboard I have a few general insights I want to dump here.
Most dashboarding frameworks and software can be divided based on how they manage the space on the screen/paper.
Dashboards are nearly always used at full width, often fullscreen.
I've seen a few projects try to get dashboard developed as a content for a page fit into a fullscreen layout, and it gets messy quite easily.
Developing a dashboard on a 1080p display and showing it in a 4k display is not uncommon nowadays. 4k TVs are very cheap, 4k monitors still expensive and uncommon.
A few dashboard framework just display gigantic widget on such screens and do not use all the available resolution.
Widgets in a dashboard are useless without a title or description of sorts. The appearance and position of the title should be standardized to be easy to find. A good example is the Mozaik framework http://mozaik.herokuapp.com/
Many popular data-science graph utility output static images, and their width/height ration can be changed only in the cell code. This is an excellent reason alone to use elements of a fixed proportion, this way the user is not stuck in a loop between moving the widgets around changing their content height/width ratio.
Dashboards by their nature have a lot of empty space and generally require dark low-contrast colors to not tire the eye too much.
In a dashboard layout, modifications should be made to the markdown renderer to render the h1 and h2 elements aligned to the center.
Where to write code
I think using the new layout of jupyterlab the cell code should be written separately from the dashboard visualization. I find pointless trying to fit code in a cell that maybe just displays a two digit number.
I'll be sure to chime in with lessons learned from the implementation in the classic notebook here as the discussion evolves. For the moment, I'll summarize by saying that:
Kibana is quite awesome and extremely expandable with plugins.
With the addition of Timelion they added support for other sources of data. Is not pertinent in this particular case but it's a good move in the direction of supporting broader backends than elasticsearch only.
The main competitor of Kibana is Grafana, but is way way way specialized in timeseries data (exclusively).
Kibana's main developer is always experimenting around new ways of looking into data.
This is a good demo of a vanilla Kibana http://demo.elastic.co/beats/app/kibana . For some reason is missing Timelion.
A dashboard based on Kibana could be implemented as a Kibana app (one of the many in the left sidebar). Graphs and outputs could be exported into dashboards elements quite easily. The dashboarding part is already taken care of.
Both Visualize and Timelion would be a good (core) plugin to fork as a basis for a Jupyter-Kibana dashboard.
Timelion is cleaner and may be a good start. They've procrastinated a long time in implementing a multi-line editor in Timelion and i'm sure Elastic would be pretty happy having someone kicking it towards someting more serious (even if from a fork they would have to merge).
The main downside of Kibana is that it uses elasticsearch to save the user and dashboard data.
I started a comment yesterday and never hit the big green button. Here goes again!
I don't think porting the code from the classic notebook extension makes sense for the same reasons @blink1073 gave. I think some of the concepts are worth preserving based on the use I've seen.
All of the above assume the notebook-as-a-dashboard model. As @blink1073 pointed out, since Lab can show multiple notebooks at a time, and the Lab UI itself is a sort of a tabbed-based dashboard, you could imagine assembling a dashboard layout from the piece-parts of multiple notebooks. That brings interesting new technical challenges like how to save and share the layout, how to bundle the notebooks with the layout when sharing, how to ensure all the kernels for all those notebooks in a single layout are running, etc. I think Databricks used to follow this approach for their notebooks, but then landed a solution much more like the notebook-as-dashboard design followed by the jupyter/dashboards extension.
Given the plethora of options, I'm wondering if a mailing list post or simple user survey might help here. What are Jupyter users doing to build dashboards today? How would they like JupyterLab to better facilitate that workflow, if at all?
So glad to see this discussion. It gives me hope. Thank @parente for tagging me.
I'm a user of jupyter/dashboards as deployed by jupyter-incubator/dashboards_bundlers and jupyter-incubator/dashboards_server. Here is how I'm using it:
This is an application developed by scientists and used by emergency managers. An important feature is that each emergency manager using the dashboard gets her own kernel instance so that they can operate independently of one another.
My group was hipped to dashboards_server by the IBM Emerging Technologies blog. I saw it as a way to rapidly present what scientists had done in a notebook to emergency managers who can't be expected to have Jupyter and jupyter_dashboards installed.
If you would like to know more about why they can't be expected to have those things installed, that is a longer explanation that I could go into at your request. Suffice to say, emergency managers would have a hard time even downloading a notebook file and putting it in the right place. But I think they are a target audience with a lot of similarities to audiences for whom dashboards are useful in general. Turning science into operations is also an important topic in most scientific fields.
I very much hope that this concept continues to be supported somehow in Jupyterlab. Having a way to go rapidly from data exploration to zero-install sharing is a huge win for science. The app bundler makes this as easy as clicking a button in Jupyter.
That said, here are some impressions on what has been said so far:
@parente @jasongrout Yes, I'm very interested to participate in this discussion, thanks for including me.
Allowing single cells to be embedded also lowers the bar of entry for users that already rely on other (web based) applications to show their data.
It's also a good start to use dashboarding frameworks.
I'm also thinking >90% of the requests for content in the page do not require user interaction on most scenarios, and a caching mechanism would greatly reduce the load on Jupyter itself.
This could be as easy as a Nginx proxy with a custom config. It could work this way:
Probably there are Nodejs packages that can do this too without external software, but it would be not very Unix.
@cbcunc if you can display some custom CSS you could easily hide that top bar, at least when the page is fully loaded.
A big question:
Any idea on how to take advantage of JupyterLab multi-view capabilities?
referenced this issue
Feb 19, 2017
I'm stumbling upon this thread and just thought I would share my user experience as well.
I wholeheartedly agree with all that @cbcunc said. I am an enthusiastic user of the Jupyter ecosystem (in a corporate environment) but I do find that sharing my work is by far the biggest limitation of the Jupyter workflow. Static html doesn't do justice to the power of notebooks and so the whole interactive dashboard effort is critically important. In my experience a tiny fraction of the audience who would be interested in viewing dashboards is willing/able to set up an environment to render them locally. In this context I do share @cbcunc worries regarding dashboards_bundler and dashboards_server.
Regarding @parente comment on the grid layout: I personally think they are extremely useful and my guess would be people haven't really warmed up to them yet (the dashboard linked to by @cbcunc appears to use it?) .
In any case thanks all for your effort - I realise there are a lot of priorities on the jupyterlab roadmap, just wanted to say that from my user's perspective sharing should be near the top as I think it would really improve the workflow and drive engagement.
referenced this issue
Mar 24, 2017
Pretty late to the party, having been pulled on a bunch of other not-directly-jupyter-related things for the last couple months. I'm still quite excited about the prospects of dashboard building and viewing with/from notebooks.
First off, I'll take issue to "dashboard" being the thing we want at the end of the day... or the only thing we want. Slides, sites, posters, etc. all basically require solving the same problem of creating a visual artifact, static or live, that reflects some work that is done, and we can build all of them with the same, slowly-revealed UI.
Things in JupyterLab that will make things less bad than the were when we built RISE, nbpresent and dashboards:
Macro: really seems like we can solve many more problems if we get beyond the single pair of axes.
Meso: When viewing "unassisted" (i.e. i just navigated here), everybody can and will scroll, and some things are better when scrolled horizontally vs vertically.
Micro: grid vs. stacking vs. drawing vs. ???... there's certainly a place for all of them, and I started going down that road with nbpresent with customizable (or even automatic) layout engines. Though not for lack of trying, @parente and I couldn't really come up with a satisfactory way to bridge the metadata schemes from nbpresent and dashboards, but we could resurrect that idea, as we shouldn't need two things when one can do. The phosphor layouts are very good, and would cover most of the cases, or could be extended: one of the layouts I didn't get to was a constraint-based, but could really do some nice things, and might be a superset of all of the other layouts, given the right UI.
Interactivity and Events
Color, Typography and Spacing
Moving up a little, I think being able to have your current dashboard (which lives in one notebook) be able to transclude (or fake transclude, with a local copy and relative link) content from other sources would solve a tremendous number of problems. To get here, we basically need cell ids. Notebooks aside, with the
Having these connect to multiple live kernels would be amazing, even if at first there was no crosstalk between them... though it sounds like that might happen at some point.
Once you have local static content, an
Time and again, though, one of the most requested outputs is PDF, which is probably the best archival format we have. Driving towards a DOM-based rendering mechanism with
But ideally, _click this thing in Navigator or Kitematic, get
Whew! that ended up being longer than I expected. As we roll into a period of less Phosphor/Lab churn, starting to build these things will become possible and fun!
Indeed - @sccolbert answered a lot of questions while I was doing the integration!
great that this is being picked up again! @takashimokobe: I would like to see 'dashboarding' enabling the same kind of use cases which the appmode extension for Jupyter did: Turning notebooks into regular GUIs. We found workarounds to hide code cells and by modifying the toolbar plus using some stacked sidecars. But it's far from perfect. Dashboarding is focused on visual output, if I understand the concept well. We're trying to use Jupyter as an extendable data acquisition tool, allowing us to provide flexible data analysis together with data recording in a pretty unique way. (e.g. if a feature is not implemented in the GUI, simply switch to a notebook cell. Like python consoles in traditional desktop software, c.f. FreeCAD, KiCad etc.)
A thought on the state of Python interactive viz dashboards in general: Plotly Dash has stepped into the python ecosystem in place of Shiny for R. The big promise of Dash is the ability to run a lightweight open-source flask & react app anywhere, without a jupyter server to serve interactive viz, higher performance crossfiltering/brushing/streaming etc. Unfortunately, unlike Shiny which works with any interactive viz library, Dash is limited to only plotly viz, and interaction quickly becomes complex where shiny has a few tricks to simplify interactivity - eg. see Shiny 'conductors'. We also typically develop interactive viz in notebooks, then progress the code to Dash & there is effort to keep code shared between dash & notebooks.
Does the Jupyter team have the ability/vision to create something like Shiny R where you can create dashboards in notebooks & quickly progress them to standalone interactive viz dashboards, ideally in flask/react, whilst still sharing code? Maybe collaborating with the Plotly Dash team?
@nite I'm not sure it is what you want but Holoviews/Panels/Pyviz team does a great job trying to integrate things.
Holoviews integrates matplotlib/bokeh/plotly pretty well.
You can read the end of the FAQ to see how it should compare to Shiny and Dash (it still under active developpments).
My workflow is as follow.
You have librairies supported here : pyviz/panel#2
Voila is making quite some progress in this area, where it can execute any Jupyter notebook (meaning also a Julia, R, C++ or Java/JVM kernel) and render them with any template. Voila keeps the kernel alive, but does not allow arbitrary code execution.
It is still early in development, but QuantStack/voila#14 gives a good example of that is possible with voila.
We have a strong focus on ipywidgets/jupyter-widgets (plotly also uses them now) since they allow easy reuse for other languages, e.g.
For pure Python, I have experimented with https://github.com/maartenbreddels/flask-ipywidgets/ for a more light weight flask server, but this is still experimental.
Keep an eye on these projects, get involved, try it out and give us feedback, this is highly appriciated.
@takashimokobe Very cool!
While I like the preview and the cleaner UI a great deal, what about just being able to save your current dock panel as a
This could build on the workspace JSON spec a la #5903 (probably do a schema for it) such that the
A big part of this, restoring cloned outputs in workspaces, just landed in #5981. Cloned inputs would probably be necessary, as well.
For serialization, all the "parents" could be in tabs behind the dashboard content, and load everything, and have the extension fullscreen the dockpanel and reduce the tab height to 0.
Such an extension could also look for a url param, such that on binder, you could load that workspace and kick directly into "app" mode, like presentation mode, but more severe. App mode could offer going "back to the lab" where you could tweak things.
I'd also love multiple pages...
Yes, dashboards are a place to bring multiple outputs together, but it's absolutely conceivable to have a new jupyterlab extension that streams live data on a graph and can be added to a dashboard.
Yes! This is something that @takashimokobe has already been thinking about (from a design perspective). Each dashboard is a
I guess what I'm talking about is a avoiding a bunch of serializations of almost exactly the same thing. The workspaces JSON isn't really human-editable, but there's supported machinery already in lab core for working with it (python and typescript), and it kinda forces extensioneers to get on the
We had a JEP to unite the late
But they would all reuse the existing machinery, be able to use themes, share the same DOM contracts, etc. If we decide we need a "layered drawing program" layout instead of a dock panel, we land that as a primary editing ui in an extension. If we need an "annotated timeline synced to audio/video" (a la Oriole), we land that as as a primary editing ui in an extension. But all of the extensions are just working with the underlying machinery, and not inventing new formats and places to stuff them.
I just saw this extension yesterday: https://github.com/plotly/jupyterlab-dash
This is plotly's approach to dashboarding, of course, but now you can develop your dash apps in the notebook and interact with them in a separate pane in lab, which is pretty groovy.
I was about to make a binder for it, then realized it won't work on binder due to running an arbitrary port and hard-coded
It is a nice concise way of showing how to use custom comms and adding stuff to the dockpanel, however.
referenced this issue
Mar 6, 2019
Very excited about this direction and all of the ideas here. @takashimokobe I think it is really helpful to begin thinking about the interaction model and UX of how dashboards would be created in JupyterLab. There are a lot of different aspects of dashboards and I think it is super important to begin separating them out so they can be tackled in an independent manner. Our experience of building the notebook and the lessons learned there are guiding this strongly for me...
[as an aside, there is work starting on JupyterHub to introduce oauth scopes, that could make it easy to give someone view/edit authorization to a dashboard, but not code modifications. I know @SylvainCorlay has done work on this in voila, but so far (I think) separate from JHub).]
[a second aside, if layout is stored in notebook metadata, it may be more challenging to enable real time collaboration on this, as most real time systems need a well specified schema, and treat metadata as opaque JSON blobs. I am starting to see a way to handle that case, but it is more complex that having a separate well defined file format with a concrete schema]
referenced this issue
Apr 1, 2019
It would be super clutch to have simple guides / docs / examples for how to embed JupyterLab as a component into React apps and sync State JSON between them, because this critical link is poorly documented.
This docs improvement would enable JupyterLab to have great dashboards right away without extra code because then we could drop JupyterLab components into a React dashboard site and hook it up to local or remote kernels so the lab notebook can influence the state of React and React can render the results into a variety of interactive dashboard widget components. We can leverage gazillions of existing resources for React data dashboards and developers can just choose whatever visualizations they want.
We could have a parent component which makes a State json and Dispatch function using useReducer hook. The parent could pass State and Dispatch down into dashboard visualizations and into jupyterlab using either props or React Context. Then the React Dashboard and the Jupyter Notebook can share state, and you can use Python or Julia or R code to programatically control your React data visualizations, and the whole thing could work in the cloud...
TLDR: If JupyterLab can be a React component then JupyterLab Dashboarding can leverage tons of existing React.JS dashboard tools alongside the fun/powerful/interactive JupyterLab environment, but the examples of how to pull this off aren't well documented.
Somebody write a Medium article or CodeSandbox/JSFiddle example of JupyterLab in React and we are good to go!
From "drop JupyterLab components into a React dashboard." I get the impression that he means JupyterLab components.
Ideally we would like to embed the entire lab app as a single coherent chunk into a div in a react app. This would allow us to use all of jupyterlab inside a react app, to make new notebooks, change kernels, edit and run code etc. Then the react stuff can A) manage state and B) render dashboards, forms, interfaces
https://github.com/minrk/thebelab is super cool, thanks for pointing that out!
@bionicles - rather than continuing a conversation about this on a dozen somewhat unrelated issues across multiple repos, can you create a single new issue to have this conversation? That will make it much easier for you and everyone else to follow the conversation. Thanks!