Jupyter notebook manager UI that uses the new CRD

[jlewi]

Background: #1630 replace JupyterHub with a set of Cloud Native Microservices.

@kkasravi recently added a lightweight CRD for managing Jupyter notebooks. The next step would be to adapt our UI for launching/managing Jupyter notebooks to use the CRD.

[jlewi]

@vkoukis Has volunteered Arrikto to work on this next but this will likely be a non-trivial amount of work.

[vkoukis]

@jlewi we would be happy to contribute code for this.

The general idea is: What is the minimum amount of code required to expose a very thin UI, as a microservice, which will then emit instances of @kkasravi 's Notebook CRD, as implemented in #1855?

Per the discussion at https://kubeflow.slack.com/archives/C7REE0EHK/p1543787456017300, here is a draft plan. We will be keeping this up-to-date:

1. @kkasravi commits an example CRD to the repository, so we can just kubectl create -f notebook.yaml to create notebook instances.
2. We expose the UI as a self-standing microservice, using a Python app server, independently of JupyterHub [there's definitely going to be a few rough edges there, we will be iterating on this]
3. The UI microservice presents the notebook spawn form to the user
4. Submitting the form causes the UI to emit an instance of the Notebook CRD and submit it to the K8s API.

In (2)-(3), we can also take advantage of the fact that Notebooks will be K8s objects, so we will also be able to easily list/manage running notebook servers.

@jlewi You can assign this issue to @ioandr and myself so we can keep track of it.

[kkasravi]

@vkoukis this looks good. I think we should probably add e2e tests so that what jupyterhub provides is also testable by the Notebooks CRD/Controller. IE:

• an authentication provider like iap, github
  • this would need to be done within istio or ambassador and before the UI is reached, though the UI may need to deal with CORS.
• PV/PVC examples which requires volume, volumeMounts
• Container lifecycle methods like postStart
• Local filesystem operations that required uid, fs_gid
• ServiceAccountName checks

[jlewi]

/assign @vkoukis
/assign @ioandr

[jlewi]

Design doc for overall jupyter experience: http://bit.ly/kf_jupyter_design_doc

[jlewi]

@vkoukis @ioandr Do you have an update on when an initial version might be ready? I think at the meeting on Thursday you said you had a prototype?

My suggestion would be to file separate issues for some of the issues @kkasravi mentioned and deal with them in follow on issues. I think we should focus on getting the minimal version in that would allow us to begin using it instead of JupyterHub.

For example, I will file a separate follow on issue to use service account delegation as described in http://bit.ly/kf_jupyter_design_doc.

[jlewi]

@vkoukis and @ioandr Any update on this?

[ioandr]

Hi @jlewi,

sorry for us not giving a heads-up on this sooner.

We have made progress on this together with @vkoukis and @kimwnasptd and already have an initial version of the new Jupyter UI (we provide 2 screenshots below). We believe it is inline with the Jupyter design doc and would like to share it with the community for feedback, especially on UI/UX. We will open a WIP PR asap, since we need to tackle some more issues, hopefully by the end of this week.

Btw, you can assign #2271 to me and @kimwnasptd, as soon as he becomes a member of the Kubeflow organization.

[New Jupyter UI uses Material Design Lite, just like the CentralDashboard]

[jlewi]

@vkoukis UI looks great. Is the notebook login screen going away? I thought in the initial PR the UI would just use a service account with suitable permissions to create notebooks.

[ioandr]

@vkoukis UI looks great. Is the notebook login screen going away? I thought in the initial PR the UI would just use a service account with suitable permissions to create notebooks.

Our initial approach was to require Users to login into the new Jupyter UI with the JWT that is associated with their ServiceAccount, so that the new Jupyter web app could leverage K8s Impersonation for all API requests it performs. Since the auth login UI has already been merged, we can skip the login and Impersonation parts for now. Instead, a privilleged SA will be created for the new Jupyter UI.

[jlewi]

@kimwnasptd @ioandr What is the remaining work on this issue?

[ioandr]

@kimwnasptd @ioandr What is the remaining work on this issue?

This issue is covered by #2357 and can be closed.

[vkoukis]

@jlewi PR #2357 merged the new jupyter-web-app.

Let's close this and have a new, narrower issue to keep track of the work to integrate the new web app into a unified, authenticated flow, and have users spawn notebooks in different namespaces.