Lock down jupyter pods

[jlewi]

What would it take to lock down individual Jupyter pods so that only the owner has access?

I think the use case would be you have multiple data scientists sharing a cluster. Each data scientist has access to slightly different datasets. So they want to use their own credentials (and not a shared service account). So we'd like the pods to be sufficiently locked down that they feel comfortable putting their credentials in the pod (e.g. gcloud auth login) and that except for a cluster admin, no one will be able to access the pod and grab their credentials.

I think this might look like the following

• Use a sidecar in the Jupyter pod to check the identity of the request (e.g. by looking at the JWT) and only accept requests from the owner of that server
• Use RBAC to lock down the pod.

Background on running jupyter notebook containers as root vs. non root

• Initially Jupyter didn't run as root and this caused problems when attaching extra volumes see Enable PVCs for per-user persistent storage #22 (comment)
• @willb pointed out that running containers as root doesn't work with OpenShift

[yuvipanda]

The jupyter pods are already are authenticated with internal OAuth between the Hub process and the user pods, so other users on the network shouldn't be able to access the notebook pod without authentication. We also default to disabling service account mounting in the spawned user pod to prevent users from accessing the kubernetes API. I agree that we should have easier ways to provide rolebindings for each user's pod if we wanna allow access to the k8s API - there's ongoing work in jupyterhub/kubespawner#115 to make that possible. Me and @foxish also discussed this in person at kubecon and believe we have a reasonable way forward.

…

On Jan 7, 2018 3:07 PM, "Jeremy Lewi" ***@***.***> wrote: What would it take to lock down individual Jupyter pods so that only the owner has access? I think the use case would be you have multiple data scientists sharing a cluster. Each data scientist has access to slightly different datasets. So they want to use their own credentials (and not a shared service account). So we'd like the pods to be sufficiently locked down that they feel comfortable putting their credentials in the pod (e.g. gcloud auth login) and that except for a cluster admin, no one will be able to access the pod and grab their credentials. I think this might look like the following - Use a sidecar in the Jupyter pod to check the identity of the request (e.g. by looking at the JWT) and only accept requests from the owner of that server - Use RBAC to lock down the pod. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#101>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAB23hpkUzvs5ps5QB5NTncHS8Ad4rsaks5tIU4agaJpZM4RV0yg> .

[jlewi]

I think we will definitely need access to the K8s API because we want to allow users to create other K8s resources (e.g. TfJobs) from their notebooks. Although arguably they should be using user credentials for this and not a service account.

[yuvipanda]

jupyterhub/kubespawner#110 is partially for dealing with directly mounting OAuth credentials and what not from your login provider into the pod as a secret, which might help (if you are using google auth to log in too)

We don't mount any service accounts by default, but you can certainly enable mounting :)

We'll probably also add NetworkPolicy support to the helm chart at some point, locking down traffic to only be along the paths necessary by default. Specifically, only allow traffic between hub/proxy and pods, and disallow it between

[jlewi]

jupyterhub/kubespawner#115 would allow us to spawn notebooks in their own namespace which should be a big step forward with providing a good/multi-user security model with K8s/Kubeflow.

We might want to consider contributing to that to solve our problems.

[inc0]

some good discussion https://kubeflow.slack.com/archives/C7REE0EHK/p1519842848000674?thread_ts=1519837538.000362&cid=C7REE0EHK

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.