feat: Add lifecycle hooks to VSCode and Jupyterlab Workspaces to provide persistence of conda, pip, and extensions.

[aleksandrmelnikov]

What this PR does:
Adds a design document of the implementation.

• Particularly, the background and explanation of the solution.

Adds migrations to add lifecycle hooks into the workspace templates.

Which issue(s) this PR fixes:

Fixes #623

Special notes for your reviewer:

Checklist

Please check if applies

• I have added/updated relevant unit tests
• I have added/updated relevant documentation

Required

• I accept to release these changes under the Apache 2.0 License

[rushtehrani]

@aleksandrmelnikov getting this error when running the migration. Same error when I directly copy/paste the template:

FATA[0004] Failed to run database go migrations: failed to run Go migration "20201008153033_update_jupyter_lab_template.go": error converting YAML to JSON: yaml: line 21: found character that cannot start any token 

[aleksandrmelnikov]

Please don't merge this yet, need to move the docs to another repo.

[rushtehrani]

It looks like using this approach for restoring pip/conda packages is adding an extra 10 mins delay (with JupyterLab). Need to verify whether this is because conda is taking a long time to restore or if it's somewhere else.

[rushtehrani]

It seems like conda env update attempts to update all packages, even if they exist.

There is an open issue for this: conda/conda#10218

[aleksandrmelnikov]

Good find on the "update all packages".
I'm wondering if instead of restoring the environment, we can add some logic to compare the packages.

• If it exists, skip. If it doesn't install?

[rushtehrani]

I'm wondering if instead of restoring the environment, we can add some logic to compare the packages.

Possibly, or if there is a way to only export user installed packages...

[aleksandrmelnikov]

@rushtehrani Note the change in the migration to jupyterlab. This should fix the long start time for it.

Time command to debug postStart hook:
Jupyterlab

condayml="/data/.environment.yml";
          jupytertxt="/data/.jupexported.txt";
          if [ -f "$condayml" ]; then { time conda env update -f $condayml;} 2>> /data/time.txt; fi;
          if [ -f "$jupytertxt" ]; then { time cat $jupytertxt | xargs -n 1 jupyter labextension install --no-build;} 2>> /data/juptime.txt; fi;

VScode

            condayml="/data/.environment.yml";
            vscodetxt="/data/.vscode-extensions.txt";
            if [ -f "$condayml" ]; then { time conda env update -f $condayml;} 2>> /data/time.txt; fi;
            if [ -f "$vscodetxt" ]; then { time cat $vscodetxt | xargs -n 1 code-server --install-extension;} 2>> /data/juptime.txt; fi;

[aleksandrmelnikov]

All issues should be resolved.

[rushtehrani]

Getting the message below after resuming a paused JupyterLab workspace.

I'm not sure if the --no-build flag is the way to go if this is going to happen... I did click "rebuild" in this case and it only took about 2 mins to rebuild and another 20 seconds restart. Is the rebuild triggered by UI any different than the command line version?

[aleksandrmelnikov]

@rushtehrani I was able to recreate this error and I pushed up a fix.
I tested with the following packages.

conda install nbresuse 
jupyter labextension install --no-build jupyterlab-topbar-extension jupyterlab-system-monitor

The time to install and build seems to be ~5-6 minutes.

root@jup2-0:/data# tail jupbuild.txt 
[LabBuildApp] JupyterLab 2.2.5
[LabBuildApp] Building in /opt/conda/share/jupyter/lab
[LabBuildApp] Building jupyterlab assets (build:prod:minimize)

real	2m37.921s
user	3m25.626s
sys	0m48.704s
root@jup2-0:/data# tail time.txt 

Please update conda by running

    $ conda update -n base conda



real	2m8.134s
user	1m21.518s
sys	0m5.535s
root@jup2-0:/data# 

[rushtehrani]

Based on my testing:

• Launch with no builds: 1m 30s
• Launch with latest template (with builds): ~8m
• Launch with --minimize=False flag set: ~4m

The last option seems like the best option as far as startup times, however since it doesn't minimize the extension files, it could slow down browser load times.

Question: is there a way to save JupyterLab extensions in a different directory that is mounted vs a directory on OS disk?

[aleksandrmelnikov]

Let me take a look at changing the jupyter installation directory.

[rushtehrani]

Let me take a look at changing the jupyter installation directory.

Sounds good, if that is not option, I think our best bet is to set the --minimize flag.

[aleksandrmelnikov]

Let me take a look at changing the jupyter installation directory.

Sounds good, if that is not option, I think our best bet is to set the --minimize flag.

@rushtehrani Investigated this a bit.
Jupyter lab supports a --app-dir flag.

What this does is dictate where jupyterlab installs it's assets and extensions.

[LabBuildApp] Building in /opt/conda/share/jupyter/lab
That's our current default.

We can change that to /data/jupyterlab, or any other volume.

If we go ahead and save the extensions and jupyter to the mounted disk, we have to make sure the workspace resumes and uses that directory for jupyterlab.

• Either through --app-dir or through an environment variable JUPYTERLAB_DIR.

I'm testing if JUPYTERLAB_DIR will work on start-up.

But is your intent to save the extensions to that volume on shut-down?
And restore from that volume?

[rushtehrani]

@aleksandrmelnikov this is only going to be useful if it allows us to either bypass build or reduce the time it takes to build the extensions.

If it does, then putting it in /data/.jupyterlab would probably makes the most sense assuming we can still use /data as the working directory.

The intent is to always save the data there and not have to move it on start or shutdown.

How would this work with extensions already installed in the image. Can we separate user installed extensions?

[aleksandrmelnikov]

Per discussion here: https://onepanelio.slack.com/archives/GJ88JMG9E/p1603913216187400

We're going to go ahead and use the --minimize=False flag as a trade-off.
Using volumes will require additional steps and logic to ensure it works properly.