docker: prevent manual pin of dependencies and improve build speed #95
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…yter lab v3 anymore, causing us build problems now
…avoid too big dependency resolution
Encountered problems while solving: - package vcs-8.1-py_0 requires vtk-cdat >8.1, but none of the providers can be installed
This reverts commit 498bc8c. vtk-cdat was already part of cdat repo.
… if it prevent the downgrade
…try to pin in environment.yml as well
…ate hardcode pin in Dockerfile
…enssl significantly ``` - # - openssl 3.0.0 h7f98852_2 installed - # + openssl 1.1.1l h7f98852_0 conda-forge/linux-64 2 MB ```
|
FYI @ChaamC I am at the end of road with this jupyter-conda plugin problem, I've opened an issue on their side to get more help mamba-org/gator#170 |
Zeitsperre
reviewed
Jan 14, 2022
Closed
Not sure why cartopy-0.14.3 is selected! Previous build selected cartopy-0.20.1 and geos-3.9.1 was found! ``` Encountered problems while solving: - nothing provides geos 3.5.* needed by cartopy-0.14.3-np110py27_4 ```
Zeitsperre
reviewed
Jan 17, 2022
| # Conda was stuck at this step: | ||
| # DEBUG conda.common._logic:_run_sat(607): Invoking SAT with clause count: 2500273 | ||
| # | ||
| # Pin python=3.9 because python 3.10 cause this error: |
There was a problem hiding this comment.
Additionally, PyDAP is not Python3.10 compatible. On import:
...
File ~/miniconda3/envs/pavics-sdi/lib/python3.10/site-packages/pydap/model.py:175, in <module>
173 from six import string_types
174 import numpy as np
--> 175 from collections import OrderedDict, Mapping
176 import warnings
178 from .lib import quote, decode_np_strings
ImportError: cannot import name 'Mapping' from 'collections' (/home/me/miniconda3/envs/pavics-sdi/lib/python3.10/collections/__init__.py)
|
I pinned shapely to the old 1.7.1 version and there is a new failure with the homepage nb 5: http://jenkins.ouranos.ca/job/PAVICS-e2e-workflow-tests/job/prevent-manual-pin-of-dependencies/14/consoleFull A good news is the bokeh/holoviews performance problem seems to be fixed in the new build. This new build is deployed as "beta" image in prod. |
|
I'll take a look. |
Sample error:
```
_______ pavics-sdi-master/docs/source/notebooks/forecasts.ipynb::Cell 2 ________
Notebook cell execution failed
Cell 2: Cell outputs differ
Input:
import hvplot.xarray
mtl = ds.sel(lon=45.5, lat=360 - 73.5, method="nearest")
mtl.pr.hvplot(x="time", width=300) + mtl.tas.hvplot(x="time", width=300)
Traceback:
mismatch 'application/vnd.holoviews_load.v0+json'
assert reference_output == test_output failed:
'\n(function(...\n}(window));' == '\n(function(...\n}(window));'
Skipping 1273 identical leading characters in diff, use -v to show
, now());
+ root._bokeh_is_loading = css_urls.length + js_urls.length + js_modules.length;
function on_load() {
root._bokeh_is_loading--;
if (root._bokeh_is_loading === 0) {
console.debug("Bokeh: all BokehJS libraries/stylesheets loaded");
run_callbacks()
}
}
( ... )
```
|
@ChaamC FYI the jupyter-conda plugin is fixed by uninstalling mamba from the image ! |
This was referenced Jan 22, 2022
tlvu
added a commit
to bird-house/birdhouse-deploy
that referenced
this pull request
Jan 27, 2022
Deploy new Jupyter env from PR Ouranosinc/PAVICS-e2e-workflow-tests#95 on PAVICS. Detailed changes can be found at Ouranosinc/PAVICS-e2e-workflow-tests#95. Relevant changes: ```diff < - xclim=0.31.0=pyhd8ed1ab_0 > - xclim=0.32.1=pyhd8ed1ab_0 < - ravenpy=0.7.5=pyhff6ddc9_0 > - ravenpy=0.7.8=pyh8a188c0_0 < - python=3.7.12=hb7a2778_100_cpython > - python=3.8.12=hb7a2778_2_cpython < - vcs=8.2.1=pyh9f0ad1d_0 < - numpy=1.21.4=py37h31617e3_0 > - numpy=1.21.5=py38h87f13fb_0 < - xarray=0.20.1=pyhd8ed1ab_0 > - xarray=0.20.2=pyhd8ed1ab_0 < - rioxarray=0.8.0=pyhd8ed1ab_0 > - rioxarray=0.9.1=pyhd8ed1ab_0 < - cf_xarray=0.6.1=pyh6c4a22f_0 > - cf_xarray=0.6.3=pyhd8ed1ab_0 < - gdal=3.3.2=py37hd5a0ba4_2 > - gdal=3.3.3=py38hcf2042a_0 < - rasterio=1.2.6=py37hc20819c_2 > - rasterio=1.2.10=py38hfd64e68_0 < - climpred=2.1.6=pyhd8ed1ab_1 > - climpred=2.2.0=pyhd8ed1ab_0 < - clisops=0.7.0=pyh6c4a22f_0 > - clisops=0.8.0=pyh6c4a22f_0 < - xesmf=0.6.0=pyhd8ed1ab_0 > - xesmf=0.6.2=pyhd8ed1ab_0 < - birdy=v0.8.0=pyh6c4a22f_1 > - birdy=0.8.1=pyh6c4a22f_1 < - cartopy=0.20.0=py37hbe109c4_0 > - cartopy=0.20.1=py38hf9a4893_1 < - dask=2021.11.2=pyhd8ed1ab_0 > - dask=2022.1.0=pyhd8ed1ab_0 < - numba=0.53.1=py37hb11d6e1_1 > - numba=0.55.0=py38h4bf6c61_0 < - pandas=1.3.4=py37he8f5f7f_1 > - pandas=1.3.5=py38h43a58ef_0 ```
tlvu
added a commit
to bird-house/birdhouse-deploy
that referenced
this pull request
Jan 27, 2022
Jupyter: update Jupyter env for latest XClim, RavenPy and all packages Deploy new Jupyter env from PR Ouranosinc/PAVICS-e2e-workflow-tests#95 on PAVICS. Detailed changes can be found at Ouranosinc/PAVICS-e2e-workflow-tests#95. Relevant changes: ```diff < - xclim=0.31.0=pyhd8ed1ab_0 > - xclim=0.32.1=pyhd8ed1ab_0 < - ravenpy=0.7.5=pyhff6ddc9_0 > - ravenpy=0.7.8=pyh8a188c0_0 < - python=3.7.12=hb7a2778_100_cpython > - python=3.8.12=hb7a2778_2_cpython < - vcs=8.2.1=pyh9f0ad1d_0 < - numpy=1.21.4=py37h31617e3_0 > - numpy=1.21.5=py38h87f13fb_0 < - xarray=0.20.1=pyhd8ed1ab_0 > - xarray=0.20.2=pyhd8ed1ab_0 < - rioxarray=0.8.0=pyhd8ed1ab_0 > - rioxarray=0.9.1=pyhd8ed1ab_0 < - cf_xarray=0.6.1=pyh6c4a22f_0 > - cf_xarray=0.6.3=pyhd8ed1ab_0 < - gdal=3.3.2=py37hd5a0ba4_2 > - gdal=3.3.3=py38hcf2042a_0 < - rasterio=1.2.6=py37hc20819c_2 > - rasterio=1.2.10=py38hfd64e68_0 < - climpred=2.1.6=pyhd8ed1ab_1 > - climpred=2.2.0=pyhd8ed1ab_0 < - clisops=0.7.0=pyh6c4a22f_0 > - clisops=0.8.0=pyh6c4a22f_0 < - xesmf=0.6.0=pyhd8ed1ab_0 > - xesmf=0.6.2=pyhd8ed1ab_0 < - birdy=v0.8.0=pyh6c4a22f_1 > - birdy=0.8.1=pyh6c4a22f_1 < - cartopy=0.20.0=py37hbe109c4_0 > - cartopy=0.20.1=py38hf9a4893_1 < - dask=2021.11.2=pyhd8ed1ab_0 > - dask=2022.1.0=pyhd8ed1ab_0 < - numba=0.53.1=py37hb11d6e1_1 > - numba=0.55.0=py38h4bf6c61_0 < - pandas=1.3.4=py37he8f5f7f_1 > - pandas=1.3.5=py38h43a58ef_0 ```
This was referenced Mar 18, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Overview
Previously, when xclim and ravenpy were pinning their own dependencies, the pins were ignored and we had to manually repeat the same pins again. See comment #94 (comment).
This PR allows xclim and ravenpy to manage their own dependencies pinning transparently during this Jupyter env rebuild.
Also fixed a long standing build performance along the way. Build time went from 50 mins to 25 mins and builds on DockerHub works again (fixes #51).
Deployed as "beta" image on https://pavics.ouranos.ca/jupyter for testing.
Changes
Switched to using mamba instead of conda since mamba dependency solver is faster. Mamba solver being faster at the expense of less precision so had to pin latest xclim and ravenpy to avoid random downgrade in the 2nd build phase.
Both solvers performance seem to drop exponentially when less packages are specified directly, leading to more work for the solver to discover them. Less packages specified directly because we removed all direct dependencies of xclim and ravenpy from the
environment.ymlfile.Switched to using 2 stages conda env build as another performance work-around. One single
conda env create -f /environment.ymlwas taking many days ! mamba was not much better in one single stage build.With 2 stages build, a build using conda solver takes 5 hours while mamba solver takes 25 minutes !
Reduced the number of "build layers" by merging several of them, for another small build performance gain.
jupyterlab-topbar-textandjupyterlab-theme-togglejupyterlab extension was removed due to javascript build problem. The topbar text was pretty useless. Hopefully the theme toogle is not so widely used.Had to hardcode the commit of the https://github.com/jupyter/docker-stacks repo where we get the startup script from because the latest version of those scripts are breaking us. This will have to be solve later.
Removed
vcslibrary fromcdatchannel in order to move to python 3.9. Otherwise we are stuck on 3.7 and xarray will drop 3.7 soon. I've opened an issue on CDAT side Unable to install vcs conda package in a python 3.9 environment: package vcs-8.1-py_0 requires vtk-cdat >8.1, but none of the providers can be installed CDAT/vcs#457.vcslibrary was needed to run ESGF notebooks at https://github.com/ESGF/esgf-compute-api/tree/devel/examplesRelated Issue / Discussion
Related issues Unable to run 'jupyter lab build' with jupyterlab 3.2.5, previous 3.2.4 was working fine jupyterlab/jupyterlab#11726
Notebook fix needed fix notebook 5 - take 2 PAVICS-landing#42
Matching PR to deploy this new Jupyter env to PAVICS Jupyter: update Jupyter env for latest XClim, RavenPy and all packages bird-house/birdhouse-deploy#234
Additional Information
Screenshot of UI change showing
jupyterlab-topbar-textandjupyterlab-theme-togglejupyterlab extension removed:Relevant changes:
conda env export:211123-update211216-211221-conda-env-export.diff.txt
211221-220116.1-conda-env-export.diff.txt
211123-update211216-220116.1-conda-env-export.diff.txt
220116.1-220121-conda-env-export.diff.txt
211123-update211216-220121-conda-env-export.diff.txt
conda env export:211221-conda-env-export.yml.txt
220116.1-conda-env-export.yml.txt
220121-conda-env-export.yml.txt