Jupyter docker: new full build with latest of almost everything except xclim and ravenpy to smooth transition #121
Conversation
Should be fixed by intake/intake-esm#569. Fixes #109
…dmesh() is fixed See holoviz/hvplot#1073
Anything outside of /notebook_dir/writable-workspace/ is not persisted on disk. So we make /notebook_dir/ read-only to avoid user accidentally saving valuable data there and lose it. Need to pre-create /notebook_dir/writable-workspace/ and /notebook_dir/pavics-homepage/ because Jenkins will write there so these folders are writable. See https://github.com/Ouranosinc/PAVICS-landing/blob/f733612c9805bbd46da7d8204497553d891ffd1b/content/notebooks/climate_indicators/setup_dirlayout.sh#L14-L27
Otherwise the new install did not install anything because it's the same version.
See docker/saved_buildout/conda-env-export.yml.
Using the latest 0.4x series, hopefully it is compatible.
```
Could not solve for environment specs
The following packages are incompatible
├─ jupyterlab-git 0.50.0 is installable with the potential options
│ ├─ jupyterlab-git 0.50.0 would require
│ │ └─ jupyterlab >=4,<5 , which can be installed;
│ └─ jupyterlab-git 0.50.0 conflicts with any installable versions previously reported;
└─ jupyterlab-topbar is not installable because there are no viable options
├─ jupyterlab-topbar 0.6.1 would require
│ └─ jupyterlab >=3.0.0,<4 , which conflicts with any installable versions previously reported;
└─ jupyterlab-topbar 0.6.1 would require
└─ jupyterlab >=3.0.0rc10,==3.* , which does not exist (perhaps a missing channel).
The command '/bin/sh -c umask 0000 && pip uninstall -y ravenpy && pip install --no-cache-dir git+https://github.com/CSHS-CWRA/RavenPy.git@1977732c8dbf39ea7d563e7e30052707ba8fb2ec && mamba install -c conda-forge -c cdat -c bokeh -c plotly -c pyviz/label/dev -c defaults -n birdy jupyterlab-git==0.50.0 && mamba clean --all --yes' returned a non-zero code: 1
```
tlvu
changed the title
To fix notebook 02_Extract_geographical_watershed_properties.ipynb.
…n HydroShare_integration failure
…termittent opendap.ipynb failure
fixes pavics-sdi bugs noted here : Ouranosinc/pavics-jupyter-env-issues#2, Ouranosinc/pavics-jupyter-env-issues#6 Fix for Ouranosinc/PAVICS-e2e-workflow-tests#121
fix issues related to Ouranosinc/pavics-jupyter-env-issues#2, Ouranosinc/pavics-jupyter-env-issues#6 Fix for Ouranosinc/PAVICS-e2e-workflow-tests#121 Replace #68
In notebook 02 using the PAVICS beta image `stats_resp.get(asobj=True)` returns an `xr.Dataset` ... plotting requires accessing the `band_data` variable See Ouranosinc/pavics-jupyter-env-issues#7 Note this will break the notebook when using `current` image. Do not merge until all fixes are ready Fix for Ouranosinc/PAVICS-e2e-workflow-tests#121.
|
@tlogan2000 The new Jupyter env is live. You can send announcement to our users to restart their server to get the new version. The previous "stable" version is now the new "previous" version. I've also added a "last-py38-stable-version" to preserve our last Python 3.8 env in case our user need an even older version to reproduce their work.
|
|
We need to cull a few of those entries. |
Well Previously we deploy those test versions on our staging host but since the staging host do not have all the data, we could not fully test those environments. And the only 100% sure way to not have any surprise is to test directly on the production host ! I've seen case where the JupyterHub is unable to launch the JupyterLab server ! Having the test env on the production host eliminate this issue. All the app specific envs are for our apps to reproduce the exact env of their runtime. Not all of them, as you can see, are on the "Stable" image. Since their have their own version, they are not force to update at the same time we release a new Stable version. Just ensuring all the notebooks passed is way too labor intensive, I do not want to add all the apps to the certification list ! The So that's the reason for all those multiple entries. |
Oh forgot to add that we are trying to release more often so we are continuous trying to test newer versions. Those |
Overview
This new full build has latest of almost everything except
xclimandravenpyas intermediate step to smooth transition topandas2.2 freq strings changes.Changes
New: save conda env export, DockerHub build logs and Jenkins test result in the repo to track changes much more easily between releases
Jenkins: add
SAVE_RESULTING_NOTEBOOK_TIMEOUTfor slow notebooks or slow machineJupyter env changes:
conda-packso we can export the conda env outside of the docker image if need to run locally without dockerTest
HydroShare_integration.ipynbfailing: https://github.com/Ouranosinc/PAVICS-e2e-workflow-tests/blob/931cfc924a147d07b59e88badff9f170e852a03b/docker/saved_buildout/jenkins-buildlogs-raven.txtRelated Issue / Discussion
Matching notebook fixes:
band_datavariable CSHS-CWRA/RavenPy#356popmean.shape[axis]must equal 1.) PAVICS-landing#65Deployment to PAVICS: Jupyter env: new full build with latest of almost everything bird-house/birdhouse-deploy#453
Jenkins-config changes for new notebooks: PR url: None
Other issues found while working on this one
--sanitize-withoption seems to be behaving weirdly with "newly computed (test) output" computationalmodelling/nbval#204duplicatemethod does not work as expected CSHS-CWRA/RavenPy#361Previous release: PR Jupyter docker: incremental build to get latest unreleased threddsclient for Thredds 5 compat #134
Additional Information
Full diff conda env export:
81deb99...931cfc9#diff-e8f2a6a53085ae29bb7cedc701c1d345a330651ae971555e85a5c005e94f4cd9
Full new conda env export:
https://github.com/Ouranosinc/PAVICS-e2e-workflow-tests/blob/931cfc924a147d07b59e88badff9f170e852a03b/docker/saved_buildout/conda-env-export.yml
DockerHub build log
https://github.com/Ouranosinc/PAVICS-e2e-workflow-tests/blob/931cfc924a147d07b59e88badff9f170e852a03b/docker/saved_buildout/docker-buildlogs.txt