Skip to content
This repository has been archived by the owner on Aug 30, 2022. It is now read-only.

Deployment Roadmap

Jump to bottom
Peter Parente edited this page May 24, 2016 · 30 revisions

May 2016 Update

Per the roadmap discussion we had at the Jupyter dev meeting in March:

In addition, we have opened the https://github.com/jupyter-incubator/dashboards_setup repo with docker-compose, Cloud Foundry, and tmpnb recipes for deploying the dashboard related assets for use beyond development.

March 2016 Update

We've completed most of the original roadmap as planned. At the moment we have:

The last piece has matured to the point where:

  1. Alice can write notebook using ipywidgets and lay it out as a dashboard
  2. Alice can one-click deploy the notebook to a pre-configured dashboard server
  3. Bob can login to the dashboard server and interact with Alice's notebook as a dashboard
  4. Bob cannot see Alice's code or send arbitrary code to a kernel powering Alice's dashboard
  5. Alice can update and one-click redeploy her notebook

See https://gist.github.com/parente/527cea0481afe9fabbcd for a demo setup that brings all of the above pieces together. Here's an animation created using that demo setup.

The basic building blocks of this setup appear are depicted here:

Diagram of components for deploying standalone dashboards

With additional detail added:

Detailed diagram of components for deploying standalone dashboards

Next Steps

Along the way we learned more about common deployment scenarios and aligned more with the direction Jupyter is headed with respect to interactivity both inside and outside the notebook. Here's our primary focus for continuing to move the dashboard deployment work forward, subject to change based on user requirements:

December 2015

Background

The latest stable jupyter_dashboards PyPI release 0.2.0 supports:

  • Dashboard layout mode for arranging notebook cell outputs in a grid-like fashion
  • Dashboard view mode for interacting with an assembled dashboard within the Jupyter Notebook
  • Ability to share notebooks with dashboard layout metadata in them with other Jupyter Notebook users
  • Ability to nbconvert a notebook to a separate dashboard web application

This roadmap focuses on improving the security, scalability, and compatibility story of the fourth bullet: notebooks deployed as standalone dashboard applications.

Goals

  1. Get to a stable set of PyPI releases that implement the initial countermeasures described in the [Deployed Dashboard Threat Analysis](Deployed Dashboard Threat Analysis) in support of the user stories below on Jupyter Notebook 4.x.
  2. Take stock of where the Jupyter Lab (5.0+) implementation stands and synchronize future plans for dashboards with that work.

Definitions

First, some lingo used throughout this document.

  • Web app frontend - The HTML/CSS/JavaScript that runs in the dashboard user's web browser
  • Web app backend - The web server that returns the dashboard frontend assets (e.g., a NodeJS app)
  • Kernel cluster - The web service that provides an API for requesting a kernel gateway (e.g., tmpnb, binder)
  • Kernel gateway - The web service that provides an API for requesting a kernel and communicating with it (e.g., jupyter-incubator/kernel_gateway)
  • Kernel - The compute engine that executes dashboard backend code and publishes responses to listeners (e.g., ipykernel, IRkernel)

These terms represent "slots" in a solution that enables secure, standalone dashboards. The choice of components that fill these slots (e.g., tmpnb vs binder, IPython kernel vs IRkernel, NodeJS backend vs PHP backend) will vary across use cases. The choice of how to deploy these components (e.g., kernel gateway in Docker containers vs Rocket containers, tmpnb on Swarm vs single host, use of kernel cluster for cross host scaling vs single kernel gateway only) will vary as well.

User Stories

Ability to Deploy Dashboards within Jupyter Notebook

Alice installs the jupyter_dashboards, jupyter_contentmanagement, and jupyter_dashboards_bundlers extensions for Jupyter Notebook 4.x. She then creates a new notebook. She includes various interactive widgets based on ipywidgets in the notebook. She gives the notebook a dashboard layout. She clicks File → Deploy as → Local Dashboard in Jupyter. A new browser tab opens and redirects to a /files/local_dashboards URL of her Jupyter server. She sees her notebook as a standalone dashboard application. She can share the URL of the dashboard with anyone that has access to her Jupyter instance. She can see and manage running instances of kernels associated with the dashboard in the Running tab on the Jupyter Notebook main page.

Alice's deployment looks like the following in the end:

Ability to Download a Dashboard Node App Bundle

Bob installs the jupyter_dashboards, jupyter_contentmanagement, and jupyter_dashboards_bundlers. He then creates a new notebook. He includes various interactive widgets based on ipywidgets in the notebook. He gives the notebook a dashboard layout. He clicks File → Download as → Dashboard Node App (.zip). A new browser tab opens and a zip download commences. Bob opens the zip file and sees a README, Dockerfile, Makefile, and other assets needed to build, configure, and deploy a NodeJS dashboard backend. Bob uses any nconf supported mechanism to configure his app in the following manner:

  • HTTPS on all connections from browsers
  • Require shared username/password login to access the dashboard web app
  • URL of the kernel cluster that will provision and scale kernel gateways for the dashboard
  • API keys that allow the dashboard backend to request kernel gateways and kernels

Bob then manually deploys his app to his Docker host of choice. In the end, Bob's deployment looks like the following:

The choice of web host, notebook host, and kernel cluster provider are all left up to Bob.

Ability to Write a Bundler that Deploys Dashboard Apps to Cloud Host acme.com

Cathy works at acme.com. Her company offers cloud hosting for Jupyter Notebook server, web apps, and Jupyter kernels (i.e., kernel cluster + kernel gateway + kernels). She wants to make it easy for users of jupyter_dashboards to deploy their notebooks as standalone dashboard web applications at acme.com. Cathy writes a new plugin for jupyter_cms that:

  1. Adds a new Deploy as → Dashboard on acme.com menu item
  2. Handles a click on that menu by:
    • Turning the notebook into a web app for deployment at acme.com
    • Configuring the app with the URL for the acme.com kernel cluster, API keys to allow the app to access the kernel cluster, specifics about the kernel gateway it needs to request to support the notebook code, etc.
    • Pushing the app bundle to the acme.com web host
    • Returning the public URL of the deployed dashboard

Cathy's acme.com solution looks like the following in the end. Note that it provides one potential solution to Bob's web, notebook, and kernel cluster service needs in the scenario above.

Work for Goal #1

The list below is roughly sequenced, with some activities happening in parallel.

Issues

  • How do common JS assets get shared among dashboard layout extension, NodeJS dashboard backend, static web app for local deployment, etc. after the refactoring?
    • Short term: duplication with JS in jupyter-incubator/dashboards serving as the master copy
    • Long term: move the JS bits to a common package
  • How are prerequisites like libraries, kernels, and data used by notebooks satisfied upon deployment?
    • Using only the freely available, open source assets: manually.
    • In a "walled garden" solution where the notebook environment, kernel compute cluster, docker registry, package manager, and data store are all known: in an automated fashion by custom bundlers.

Work for Goal #2

Largely unknown, but definitely predicated on:

  • Agreement with the community on what "dashboards" mean within the grander workbench context.
  • Exploration of how to port relevant features from the current implementations to Phosphor plugins.

h8i

Clone this wiki locally