'ks delete ${KF_ENV} -c kubeflow-core' doesn't take down user notebook pods

[amygdala]

Running: ks delete ${KF_ENV} -c kubeflow-core apparently does not delete the user notebook pod(s) (though tf-hub-0 itself is deleted properly).

I couldn't find any existing issues mentioning this, but maybe I didn't use the right search keywords... as I assume this is not intended? Is it a ksonnet-related thing?

I am using version v0.1.3.

[jlewi]

I think this is working as intended.

The Jupyter pods aren't created by the ksonnet component they are created by the JupyterHub pod spawner.

So if we wanted to delete them when JupyterHub is deleted then the spawner should probably set the owner reference on the notebooks pods. We could file an issue against the spawner.

However, I'm not sure that behavior is desired. Deleting notebook pods could potentially result in loss of work so I think it should be a very explicit action.

If you wanted to delete everything including the pods you could do so by deleting the namespace.

What do others think

/cc @pdmack

[amygdala]

a n00b question: could one get to the notebook in a browser after the tf-hub & ambassador stuff is torn down? It might more be that you would instead just need to ssh in to the pod container and recover files?

[pdmack]

@jlewi is correct. The ksonnet app doesn't know about the arbitrary Jupyter notebook that was launched from JH, so it won't be cleaned up by ks.

@amygdala Yes, you could possibly exec to the pod but I think if you are concerned about preserving NB content you should enable the jupyterNotebookPVCMount parameter for /home/jovyan instead.

[amygdala]

@pdmack That makes sense; I was more asking from the viewpoint of one of our users than specific concerns of my own.
That is, assuming the notebook pods stay up for some indefinite amount of time (the amount of time I guess is up to the cluster admin...) -- how do we best document in the user guide what a given user does to get to their stuff under those circs?

(A full user guide is a ways away, but it's good to keep this kind of thing in mind for when we get to the point where usability moves more to the forefront).

[jlewi]

@amygdala They should be able to access their pods via JupyterHub which will be accessible via the central UI once #810 is fixed. If the user already launched a notebook for themselves than the JupyterHub UI will show this and allow users to either login into it or stop it.