Enable PVCs for per-user persistent storage #22

yuvipanda · 2017-12-08T05:14:52Z

Remember that tearing down a cluster doesn't clean up the PVCs
provisioned, and you have to actually delete the PVCs before
tearing down the cluster manually.

This relies on a default storage provisioner being present
on the k8s cluster. On GKE, this gives you 10Gi of stnadard
non-ssd storage.

Remember that tearing down a cluster doesn't clean up the PVCs provisioned, and you have to actually delete the PVCs before tearing down the cluster manually. This relies on a default storage provisioner being present on the k8s cluster. On GKE, this gives you 10Gi of stnadard non-ssd storage.

jlewi · 2017-12-08T18:23:37Z

I tried this and the PVC doesn't appear to get attached to the pod for the notebook.
Am I missing something?

I tried looking at https://github.com/jupyterhub/kubespawner/blob/master/kubespawner/spawner.py and I see where the claim gets created but I don't see where its attached to the pod.

yuvipanda · 2017-12-08T18:27:25Z

Ah, right. we've to set the volumes too explicitly, see https://github.com/jupyterhub/zero-to-jupyterhub-k8s/blob/master/images/hub/jupyterhub_config.py#L79 for the extra bits needed. Specifically c.KubeSpawner.volumes and c.KubeSpawner.volume_mounts

yuvipanda · 2017-12-08T18:29:19Z

I updated it!

jlewi · 2017-12-08T20:53:21Z

This works. Thanks for the quick fix.

foxish · 2017-12-08T22:37:42Z

We should add instructions for it to continue to work on Minikube before merging. (or verify if it already does work)

yuvipanda · 2017-12-08T22:39:22Z

This should work on minikube, it ships with a default hostPath-based provisioner.

foxish · 2017-12-08T22:46:24Z

Ah, sounds good. As far as I know, that's an addon, but I guess it's enabled by default. We should have a line in the docs that says that we depend on the minikube addon for default-storageclass and how one can check it's state. Rest LGTM!

jlewi · 2017-12-08T22:51:54Z

Now my notebook pods seems to be hitting some issues when it tries to startup

usermod: no changes
Execute the command as jovyan
[W 2017-12-08 22:49:42.057 SingleUserNotebookApp configurable:168] Config option `open_browser` not recognized by `SingleUserNotebookApp`.  Did you mean `browser`?
Traceback (most recent call last):
  File "/opt/conda/lib/python3.6/site-packages/traitlets/traitlets.py", line 528, in get
    value = obj._trait_values[self.name]
KeyError: 'runtime_dir'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/conda/bin/jupyterhub-singleuser", line 6, in <module>
    main()
  File "/opt/conda/lib/python3.6/site-packages/jupyterhub/singleuser.py", line 455, in main
    return SingleUserNotebookApp.launch_instance(argv)
  File "/opt/conda/lib/python3.6/site-packages/jupyter_core/application.py", line 266, in launch_instance
    return super(JupyterApp, cls).launch_instance(argv=argv, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/traitlets/config/application.py", line 657, in launch_instance
    app.initialize(argv)
  File "<decorator-gen-7>", line 2, in initialize
  File "/opt/conda/lib/python3.6/site-packages/traitlets/config/application.py", line 87, in catch_config_error
    return method(app, *args, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/notebook/notebookapp.py", line 1294, in initialize
    self.init_configurables()
  File "/opt/conda/lib/python3.6/site-packages/notebook/notebookapp.py", line 1033, in init_configurables
    connection_dir=self.runtime_dir,
  File "/opt/conda/lib/python3.6/site-packages/traitlets/traitlets.py", line 556, in __get__
    return self.get(obj, cls)
  File "/opt/conda/lib/python3.6/site-packages/traitlets/traitlets.py", line 535, in get
    value = self._validate(obj, dynamic_default())
  File "/opt/conda/lib/python3.6/site-packages/jupyter_core/application.py", line 99, in _runtime_dir_default
    ensure_dir_exists(rd, mode=0o700)
  File "/opt/conda/lib/python3.6/site-packages/jupyter_core/utils/__init__.py", line 13, in ensure_dir_exists
    os.makedirs(path, mode=mode)
  File "/opt/conda/lib/python3.6/os.py", line 210, in makedirs
    makedirs(head, mode, exist_ok)
  File "/opt/conda/lib/python3.6/os.py", line 210, in makedirs
    makedirs(head, mode, exist_ok)
  File "/opt/conda/lib/python3.6/os.py", line 210, in makedirs
    makedirs(head, mode, exist_ok)
  File "/opt/conda/lib/python3.6/os.py", line 220, in makedirs
    mkdir(name, mode)
PermissionError: [Errno 13] Permission denied: '/home/jovyan/.local'

I'm going to dry deleting the PV, PVC

yuvipanda · 2017-12-08T22:54:46Z

Where are you running this on? It looks like the user can't write to the provisioned pv. I've forgotten how that works on GKE...

jlewi · 2017-12-08T23:15:43Z

I'm running on GKE. I'm guessing the directory is owned by root so the fact that the container is running as user joyuan is a problem.

jlewi · 2017-12-08T23:27:34Z

Yup the root directory of the PD is owned by root. I was able to modify the config so I mount the pd in a subdirectory of my home directory. I could then kubectl exec into the container to change the permissions to make it world readable.

I'm not sure how we would automate this.

foxish · 2017-12-08T23:36:34Z

Can we just run the jupyter containers in this configuration as Unix root? Every user gets a dedicated pod+pd anyway. The per-user model may make more sense in a shared NFS type of setup perhaps.

jlewi · 2017-12-08T23:38:55Z

That's one option. It looks the juptyer container runs the script start-singleuser.sh as root and then does an su to joyuan. So presumably the startup script could create needed directories and set permissions before issuing the su.

foxish · 2017-12-08T23:45:03Z

That sounds like a good idea. To either do it in the startup script, or an init container, in the interest of disallowing users from changing certain aspects of their pod. We should ensure that this is all truly portable as well.

yuvipanda · 2017-12-09T00:06:13Z

We have had good success just setting fsGid (there is a traitlet for it, I forget what it is) to the gid. It works across the clouds we have tried, and does the initial chown I think. IMO that is a better default than running as root.

…

On Dec 8, 2017 5:45 PM, "Anirudh Ramanathan" ***@***.***> wrote: That sounds like a good idea. To either do it in the startup script, or an init container, in the interest of disallowing users from changing certain aspects of their pod. We should ensure that this is all truly portable as well. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#22 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAB23lotW4-rbNFXs4dQvU8n5ZQ2hvqiks5s-cn_gaJpZM4Q6mgi> .

vishh · 2017-12-09T02:05:54Z

Yeah, the trick is to use a filesystem group in the pod spec which will be a gid that will be applied to the filesystem (sticky gid) and also get set on the container process group. Here is an example. It's unfortunate that the default behavior in GKE does not support this. Container images are not easily introspectable.

vishh · 2017-12-09T02:07:07Z

Do we have to update kubespawner to set fsGroup or is it already supported?

yuvipanda · 2017-12-09T19:47:39Z

Yep, we can set it with c.KubeSpawner.singleuser_fs_gid. What's the gid on the default image? 1000? I agree it's sad that it can't be easily introspected and has to be set explicitly.

UnionGlobal · 2017-12-10T04:34:45Z

◦	 FGBJQ

"}},705da7d71921eb85673912784f0d2c1b51],["CurrentCommunityInitialData",[],{},490],["ResourceWatcher",[],{"module":null},1404],["MPageletUtilities",[],{},358],["MJSEnvironment",[],{"IS_APPLE_WEBKIT_IOS":true,"IS_TABLET":false,"IS_ANDROID":false,"IS_CHROME":false,"IS_FIREFOX":false,"IS_WINDOWS_PHONE":false,"OS_VERSION":6.1,"PIXEL_RATIO":2,"BROWSER_NAME":"Mobile Safari"},46],["ZeroCategoryHeader",[],{},1127],["ZeroRewriteRules",[],{},1478],["UserAgentData",[],{"browserArchitecture":"32","browserFullVersion":
20799920_1500726386641166_711419247657835507_n.jpg
AeahgoYEfVcoFi1E

jlewi · 2017-12-15T15:00:03Z

I suggest closing this PR in favor of #36. #36 introduces ksonnet configs and includes the ability to attach extra volumes in the JupyterConfig. If we need to add additional options like fsGroupId we can do so in our ksonnet configs.

yuvipanda · 2017-12-16T21:00:35Z

The ksonnet stuff is great! Going to close this now :)

akshaysaxena16 · 2018-02-22T12:46:39Z

Hi,

I have followed these steps https://github.com/kubeflow/kubeflow/blob/master/user_guide.md to deploy kubeflow.
I'm able to see Jupyter notebook interface after logging into the jupyter hub.
now to add persistent volume if I follow the steps mentioned here https://github.com/kubeflow/kubeflow/pull/22/files/48ffe99d30eb31abc4d54c4553483b6c5b2034bc and make these changes in jupyter_spawner.py file.
tf-hub goes into CrashLoopBackOff error saying
IndentationError: unexpected indent

Can you suggest some solution to this to add persistent volume ?

jlewi · 2018-02-22T15:40:06Z

@akshaydtada Can you open a new issue please? @inc0 made some recent changes. I suspect an indent error in kube_spawner.py

While we were adding pvc for every jupyter instance, we didn't mount it anywhere. Let's mount it to work dir, as I assume this is dir where users will likely put their notebooks. This will ensure that work will be retained even if pod dies. Fix #19 Fix #22

… node pool. * Fix kubeflow#22; test flakes by making the GKE resource names depend on the deployment name so multiple deploymnets won't reference the same resources.

Signed-off-by: YujiOshima <yuji.oshima0x3fd@gmail.com>

[pull] master from kubeflow:master

yuvipanda mentioned this pull request Dec 8, 2017

Fault tolerant storage for Jupterhub #19

Closed

Mount the spawned PVC into the pod

48ffe99

jlewi approved these changes Dec 8, 2017

View reviewed changes

jlewi mentioned this pull request Dec 8, 2017

ksonnet configs for deploying the TfJob CRD & Controller kubeflow/training-operator#208

Closed

yuvipanda closed this Dec 16, 2017

yuvipanda deleted the user-storage branch December 16, 2017 21:03

inc0 mentioned this pull request Feb 21, 2018

Add mount of pvc #270

Merged

umar-sik mentioned this pull request Feb 25, 2018

OAuth state missing from cookies jupyterhub/jupyterhub#1682

Closed

jlewi mentioned this pull request Mar 1, 2018

Lock down jupyter pods #101

Closed

jlewi mentioned this pull request Apr 27, 2018

Can we put all of /home on PV for Jupyter notebooks #561

Closed

yanniszark pushed a commit to arrikto/kubeflow that referenced this pull request Feb 15, 2021

add test script and argo files (kubeflow#22)

830e2a7

Signed-off-by: YujiOshima <yuji.oshima0x3fd@gmail.com>

VaishnaviHire pushed a commit to VaishnaviHire/kubeflow that referenced this pull request Jul 1, 2022

Merge pull request kubeflow#22 from kubeflow/master

a09d60b

[pull] master from kubeflow:master

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enable PVCs for per-user persistent storage #22

Enable PVCs for per-user persistent storage #22

yuvipanda commented Dec 8, 2017

jlewi commented Dec 8, 2017

yuvipanda commented Dec 8, 2017

yuvipanda commented Dec 8, 2017

jlewi commented Dec 8, 2017

foxish commented Dec 8, 2017 •

edited

yuvipanda commented Dec 8, 2017

foxish commented Dec 8, 2017

jlewi commented Dec 8, 2017

yuvipanda commented Dec 8, 2017

jlewi commented Dec 8, 2017

jlewi commented Dec 8, 2017

foxish commented Dec 8, 2017

jlewi commented Dec 8, 2017

foxish commented Dec 8, 2017

yuvipanda commented Dec 9, 2017 via email

vishh commented Dec 9, 2017

vishh commented Dec 9, 2017

yuvipanda commented Dec 9, 2017

UnionGlobal commented Dec 10, 2017

jlewi commented Dec 15, 2017

yuvipanda commented Dec 16, 2017

akshaysaxena16 commented Feb 22, 2018

jlewi commented Feb 22, 2018

Enable PVCs for per-user persistent storage #22

Enable PVCs for per-user persistent storage #22

Conversation

yuvipanda commented Dec 8, 2017

jlewi commented Dec 8, 2017

yuvipanda commented Dec 8, 2017

yuvipanda commented Dec 8, 2017

jlewi commented Dec 8, 2017

foxish commented Dec 8, 2017 • edited

yuvipanda commented Dec 8, 2017

foxish commented Dec 8, 2017

jlewi commented Dec 8, 2017

yuvipanda commented Dec 8, 2017

jlewi commented Dec 8, 2017

jlewi commented Dec 8, 2017

foxish commented Dec 8, 2017

jlewi commented Dec 8, 2017

foxish commented Dec 8, 2017

yuvipanda commented Dec 9, 2017 via email

vishh commented Dec 9, 2017

vishh commented Dec 9, 2017

yuvipanda commented Dec 9, 2017

UnionGlobal commented Dec 10, 2017

jlewi commented Dec 15, 2017

yuvipanda commented Dec 16, 2017

akshaysaxena16 commented Feb 22, 2018

jlewi commented Feb 22, 2018

foxish commented Dec 8, 2017 •

edited