Helm chart does not work with default hostPath dynamic provisioner on minikube

[yuvipanda]

From @gvacaliuc

Getting an issue with the Helm chart on minikube when I don't specify a singleuse storage type of none:

singleuser:
  storage:
    type: none

How to reproduce:

Launch a new minikube cluster and use a Helm + JupyterHub configuration of this repo's minikube_config.yml without lines 7-8. If you start up a singleuser server (time is a bit sensitive since kubernetes will kill the pod after a number of failed restarts) and then rush to the command line, executing the following will get some similar error messages:

$ kubectl get pods
NAME                                READY     STATUS             RESTARTS   AGE
hub-deployment-2728205822-5qrqv     1/1       Running            0          37m
jupyter-dummy                       0/1       CrashLoopBackOff   1          7s
proxy-deployment-1227971824-cdg8z   1/1       Running            0          2h

$ kubectl describe pod jupyter-dummy > describe_pod_log.txt

$ kubectl logs jupyter-dummy 
/srv/venv/lib/python3.5/site-packages/IPython/paths.py:69: UserWarning: IPython parent '/home/jovyan' is not a writable location, using a temp directory.
  " using a temp directory.".format(parent))
Traceback (most recent call last):
  File "/srv/venv/bin/jupyterhub-singleuser", line 6, in <module>
    main()
  File "/srv/venv/lib/python3.5/site-packages/jupyterhub/singleuser.py", line 322, in main
    return SingleUserNotebookApp.launch_instance(argv)
  File "/srv/venv/lib/python3.5/site-packages/jupyter_core/application.py", line 267, in launch_instance
    return super(JupyterApp, cls).launch_instance(argv=argv, **kwargs)
  File "/srv/venv/lib/python3.5/site-packages/traitlets/config/application.py", line 657, in launch_instance
    app.initialize(argv)
  File "<decorator-gen-7>", line 2, in initialize
  File "/srv/venv/lib/python3.5/site-packages/traitlets/config/application.py", line 87, in catch_config_error
    return method(app, *args, **kwargs)
  File "/srv/venv/lib/python3.5/site-packages/notebook/notebookapp.py", line 1290, in initialize
    super(NotebookApp, self).initialize(argv)
  File "<decorator-gen-6>", line 2, in initialize
  File "/srv/venv/lib/python3.5/site-packages/traitlets/config/application.py", line 87, in catch_config_error
    return method(app, *args, **kwargs)
  File "/srv/venv/lib/python3.5/site-packages/jupyter_core/application.py", line 243, in initialize
    self.migrate_config()
  File "/srv/venv/lib/python3.5/site-packages/jupyterhub/singleuser.py", line 239, in migrate_config
    super(SingleUserNotebookApp, self).migrate_config()
  File "/srv/venv/lib/python3.5/site-packages/jupyter_core/application.py", line 169, in migrate_config
    migrate()
  File "/srv/venv/lib/python3.5/site-packages/jupyter_core/migrate.py", line 240, in migrate
    ensure_dir_exists(env['jupyter_config'])
  File "/srv/venv/lib/python3.5/site-packages/ipython_genutils/path.py", line 167, in ensure_dir_exists
    os.makedirs(path, mode=mode)
  File "/usr/lib/python3.5/os.py", line 241, in makedirs
    mkdir(name, mode)
PermissionError: [Errno 13] Permission denied: '/home/jovyan/.jupyter'

So the singleuser container is running into permissions issues writing to the /home/jovyan/ directory, which is mounted on the PersistentVolumeClaim claim-dummy, I think (see the attached file from above)? More information regarding the pvcs:

$ kubectl get pvc
NAME                      STATUS    VOLUME                                     CAPACITY   ACCESSMODES   STORAGECLASS   AGE
claim-dummy               Bound     pvc-087fcf72-8a86-11e7-9cf9-080027a064f0   1Gi        RWO           standard       1h
hub-db-dir                Bound     pvc-5016b6c8-8a7b-11e7-9cf9-080027a064f0   1Gi        RWO           standard       3h
jupyterhub-minikube-pvc   Bound     pvc-b5108598-8a85-11e7-9cf9-080027a064f0   1Gi        RWX           standard       1h

$ kubectl describe pvc claim-dummy 
Name:		claim-dummy
Namespace:	default
StorageClass:	standard
Status:		Bound
Volume:		pvc-087fcf72-8a86-11e7-9cf9-080027a064f0
Labels:		app=jupyterhub
		heritage=jupyterhub
		hub.jupyter.org/username=dummy
Annotations:	control-plane.alpha.kubernetes.io/leader={"holderIdentity":"c627a2c9-8a7a-11e7-9cf9-080027a064f0","leaseDurationSeconds":15,"acquireTime":"2017-08-26T17:43:13Z","renewTime":"2017-08-26T17:43:15Z","lea...
		pv.kubernetes.io/bind-completed=yes
		pv.kubernetes.io/bound-by-controller=yes
		volume.beta.kubernetes.io/storage-provisioner=k8s.io/minikube-hostpath
Capacity:	1Gi
Access Modes:	RWO
Events:		<none>

$ kubectl describe pv pvc-087fcf72-8a86-11e7-9cf9-080027a064f0 
Name:		pvc-087fcf72-8a86-11e7-9cf9-080027a064f0
Labels:		<none>
Annotations:	hostPathProvisionerIdentity=c627a242-8a7a-11e7-9cf9-080027a064f0
		pv.kubernetes.io/provisioned-by=k8s.io/minikube-hostpath
		volume.beta.kubernetes.io/storage-class=standard
StorageClass:	standard
Status:		Bound
Claim:		default/claim-dummy
Reclaim Policy:	Delete
Access Modes:	RWO
Capacity:	1Gi
Message:	
Source:
    Type:	HostPath (bare host directory volume)
    Path:	/tmp/hostpath-provisioner/pvc-087fcf72-8a86-11e7-9cf9-080027a064f0
Events:		<none>

The underlying issue is that the directories created by the hostPath provisioner are only writeable by root, which is problematic.

@minrk has a temporary solution here: jupyterhub/helm-chart#62 (comment)

But real solution needs work upstream.

[minrk]

Reposting my fix since helm-chart issues are going away. I have been accumulating little workarounds for dealing with wrinkles on minikube here. The one for hostpath volumes permissions:

# make hostpath volumes world-writable by default
sudo chmod -R a+rwX /tmp/hostpath-provisioner/
sudo setfacl -d -m u::rwX /tmp/hostpath-provisioner/

[yuvipanda]

kubernetes/minikube#1990 is upstream issue

[yuvipanda]

kubernetes/minikube#1991 was my upstream pull request, and it's been merged! So this should be fixed in next minikube release.

[yuvipanda]

The latest version of minikube (v0.22.3) now contains my fix, so this should be sorted.

Thank you very much for reporting, @gvacaliuc!