Skip to content

Commit

Permalink
Merge pull request #95 from Ouranosinc/prevent-manual-pin-of-dependen…
Browse files Browse the repository at this point in the history 
…cies

docker: prevent manual pin of dependencies and improve build speed

# Overview

Previously, when xclim and ravenpy were pinning their own dependencies, the pins were ignored and we had to manually repeat the same pins again.  See comment #94 (comment).

This PR allows xclim and ravenpy to manage their own dependencies pinning transparently during this Jupyter env rebuild.

Also fixed a long standing build performance along the way.  Build time went from 50 mins to 25 mins and builds on DockerHub works again.

Deployed as "beta" image on https://pavics.ouranos.ca/jupyter for testing.


## Changes

- Switched to using mamba instead of conda since mamba dependency solver is faster.  Mamba solver being faster at the expense of less precision so had to pin latest xclim and ravenpy to avoid random downgrade in the 2nd build phase.

  Both solvers performance seem to drop exponentially when less packages are specified directly, leading to more work for the solver to discover them.  Less packages specified directly because we removed all direct dependencies of xclim and ravenpy from the `environment.yml` file.

- Switched to using 2 stages conda env build as another performance work-around.  One single `conda env create -f /environment.yml` was taking many days !  mamba was not much better in one single stage build.

  With 2 stages build, a build using conda solver takes 5 hours while mamba solver takes 25 minutes !

- Reduced the number of "build layers" by merging several of them, for another small build performance gain.

- `jupyterlab-topbar-text` and `jupyterlab-theme-toggle` jupyterlab extension was removed due to javascript build problem.  The topbar text was pretty useless.  Hopefully the theme toogle is not so widely used.

- Had to hardcode the commit of the https://github.com/jupyter/docker-stacks repo where we get the startup script from because the latest version of those scripts are breaking us.  This will have to be solve later.

- Removed  `vcs` library from `cdat` channel in order to move to python 3.9.  Otherwise we are stuck on 3.7 and xarray will drop 3.7 soon.   I've opened an issue on CDAT side  CDAT/vcs#457.  `vcs` library was needed to run ESGF notebooks at https://github.com/ESGF/esgf-compute-api/tree/devel/examples


## Related Issue / Discussion

- Related issues jupyterlab/jupyterlab#11726

- Notebook fix needed Ouranosinc/PAVICS-landing#42


## Additional Information

- Screenshot of UI change showing `jupyterlab-topbar-text` and `jupyterlab-theme-toggle` jupyterlab extension removed:
![Screenshot from 2021-12-21 17-10-25](https://user-images.githubusercontent.com/11966697/147004835-ce14fc15-d04f-4956-b416-b45927b5cb4d.png)

- Relevant changes:
```diff
<   - xclim=0.31.0=pyhd8ed1ab_0
>   - xclim=0.32.1=pyhd8ed1ab_0

<   - ravenpy=0.7.5=pyhff6ddc9_0
>   - ravenpy=0.7.8=pyh8a188c0_0

<   - python=3.7.12=hb7a2778_100_cpython
>   - python=3.8.12=hb7a2778_2_cpython

# removed
<   - vcs=8.2.1=pyh9f0ad1d_0

<   - numpy=1.21.4=py37h31617e3_0
>   - numpy=1.21.5=py38h87f13fb_0

<   - xarray=0.20.1=pyhd8ed1ab_0
>   - xarray=0.20.2=pyhd8ed1ab_0

<   - rioxarray=0.8.0=pyhd8ed1ab_0
>   - rioxarray=0.9.1=pyhd8ed1ab_0

<   - cf_xarray=0.6.1=pyh6c4a22f_0
>   - cf_xarray=0.6.3=pyhd8ed1ab_0

<   - gdal=3.3.2=py37hd5a0ba4_2
>   - gdal=3.3.3=py38hcf2042a_0

<   - rasterio=1.2.6=py37hc20819c_2
>   - rasterio=1.2.10=py38hfd64e68_0

<   - climpred=2.1.6=pyhd8ed1ab_1
>   - climpred=2.2.0=pyhd8ed1ab_0

<   - clisops=0.7.0=pyh6c4a22f_0
>   - clisops=0.8.0=pyh6c4a22f_0

<   - xesmf=0.6.0=pyhd8ed1ab_0
>   - xesmf=0.6.2=pyhd8ed1ab_0

<   - birdy=v0.8.0=pyh6c4a22f_1
>   - birdy=0.8.1=pyh6c4a22f_1

<   - cartopy=0.20.0=py37hbe109c4_0
>   - cartopy=0.20.1=py38hf9a4893_1

<   - dask=2021.11.2=pyhd8ed1ab_0
>   - dask=2022.1.0=pyhd8ed1ab_0

<   - numba=0.53.1=py37hb11d6e1_1
>   - numba=0.55.0=py38h4bf6c61_0

<   - pandas=1.3.4=py37he8f5f7f_1
>   - pandas=1.3.5=py38h43a58ef_0

```

- Full diff of `conda env export`: 

[211123-update211216-211221-conda-env-export.diff.txt](https://github.com/Ouranosinc/PAVICS-e2e-workflow-tests/files/7758757/211123-update211216-211221-conda-env-export.diff.txt)

[211221-220116.1-conda-env-export.diff.txt](https://github.com/Ouranosinc/PAVICS-e2e-workflow-tests/files/7909174/211221-220116.1-conda-env-export.diff.txt)

[211123-update211216-220116.1-conda-env-export.diff.txt](https://github.com/Ouranosinc/PAVICS-e2e-workflow-tests/files/7909172/211123-update211216-220116.1-conda-env-export.diff.txt)

[220116.1-220121-conda-env-export.diff.txt](https://github.com/Ouranosinc/PAVICS-e2e-workflow-tests/files/7922627/220116.1-220121-conda-env-export.diff.txt)

[211123-update211216-220121-conda-env-export.diff.txt](https://github.com/Ouranosinc/PAVICS-e2e-workflow-tests/files/7922628/211123-update211216-220121-conda-env-export.diff.txt)

```
10,11c10,11
<   - aiobotocore=1.4.1=pyhd8ed1ab_0
<   - aiohttp=3.8.1=py37h5e8e339_0
---
>   - aiobotocore=2.1.0=pyhd8ed1ab_0
>   - aiohttp=3.8.1=py38h497a2fe_0
16,17c16,17
<   - ansi2html=1.6.0=py37h89c1867_2
<   - anyio=3.3.4=py37h89c1867_1
---
>   - ansi2html=1.6.0=py38h578d9bd_2
>   - anyio=3.5.0=py38h578d9bd_0
19,20c19,20
<   - argcomplete=1.12.3=pyhd8ed1ab_2
<   - argon2-cffi=21.1.0=py37h5e8e339_2
---
>   - argon2-cffi=21.3.0=pyhd8ed1ab_0
>   - argon2-cffi-bindings=21.2.0=py38h497a2fe_1
22c22,23
<   - async-timeout=4.0.1=pyhd8ed1ab_0
---
>   - asttokens=2.0.5=pyhd8ed1ab_0
>   - async-timeout=4.0.2=pyhd8ed1ab_0
24,25c25
<   - asynctest=0.13.0=py_0
<   - attrs=21.2.0=pyhd8ed1ab_0
---
>   - attrs=21.4.0=pyhd8ed1ab_0
30c30
<   - bcrypt=3.2.0=py37h5e8e339_2
---
>   - bcrypt=3.2.0=py38h497a2fe_2
33c33,34
<   - birdy=v0.8.0=pyh6c4a22f_1
---
>   - birdy=0.8.1=pyh6c4a22f_1
>   - black=21.12b0=pyhd8ed1ab_0
37c38
<   - bokeh=2.4.2=py37h89c1867_0
---
>   - bokeh=2.4.2=py38h578d9bd_0
40,41c41,42
<   - botocore=1.20.106=pyhd8ed1ab_0
<   - bottleneck=1.3.2=py37hb1e94ed_5
---
>   - botocore=1.23.24=pyhd8ed1ab_0
>   - bottleneck=1.3.2=py38h6c62de6_5
45,46c46,47
<   - brotli-python=1.0.9=py37hcd2ae1e_6
<   - brotlipy=0.7.0=py37h5e8e339_1003
---
>   - brotli-python=1.0.9=py38h709712a_6
>   - brotlipy=0.7.0=py38h497a2fe_1003
56c57
<   - cartopy=0.20.0=py37hbe109c4_0
---
>   - cartopy=0.20.1=py38hf9a4893_1
58,61c59,61
<   - cdms2=3.1.5=py37h84cc210_14
<   - cdtime=3.1.4=py37h7e45237_7
<   - cdutil=8.2.1=pyhd8ed1ab_1
<   - certifi=2021.10.8=py37h89c1867_1
---
>   - cdms2=3.1.5=py38hc1ef740_14
>   - cdtime=3.1.4=py38hc54c017_7
>   - certifi=2021.10.8=py38h578d9bd_1
63,64c63,64
<   - cf_xarray=0.6.1=pyh6c4a22f_0
<   - cffi=1.15.0=py37h036bc23_0
---
>   - cf_xarray=0.6.3=pyhd8ed1ab_0
>   - cffi=1.15.0=py38h3931269_0
67,68c67
<   - cftime=1.5.1.1=py37hb1e94ed_1
<   - chardet=4.0.0=py37h89c1867_2
---
>   - cftime=1.5.1.1=py38h6c62de6_1
70,71c69,70
<   - charset-normalizer=2.0.7=pyhd8ed1ab_0
<   - click=7.1.2=pyh9f0ad1d_0
---
>   - charset-normalizer=2.0.10=pyhd8ed1ab_0
>   - click=8.0.3=py38h578d9bd_1
74,75c73,74
<   - climpred=2.1.6=pyhd8ed1ab_1
<   - clisops=0.7.0=pyh6c4a22f_0
---
>   - climpred=2.2.0=pyhd8ed1ab_0
>   - clisops=0.8.0=pyh6c4a22f_0
78,82c77,81
<   - colorcet=2.0.6=pyhd8ed1ab_0
<   - configurable-http-proxy=4.5.0=node15_he6ea98c_0
<   - coverage=6.1.2=py37h5e8e339_0
<   - cryptography=35.0.0=py37hf1a17b8_2
<   - curl=7.80.0=h2574ce0_0
---
>   - colorcet=3.0.0=pyhd8ed1ab_0
>   - configurable-http-proxy=4.5.0=node16_h3b247e2_2
>   - coverage=6.2=py38h497a2fe_0
>   - cryptography=36.0.1=py38h3e25421_0
>   - curl=7.81.0=h2574ce0_0
84c83
<   - cytoolz=0.11.2=py37h5e8e339_1
---
>   - cytoolz=0.11.2=py38h497a2fe_1
86,87c85,86
<   - dask=2021.11.2=pyhd8ed1ab_0
<   - dask-core=2021.11.2=pyhd8ed1ab_0
---
>   - dask=2022.1.0=pyhd8ed1ab_0
>   - dask-core=2022.1.0=pyhd8ed1ab_0
91,93c90,92
<   - dbus=1.13.6=h48d8840_2
<   - debugpy=1.5.1=py37hcd2ae1e_0
<   - decorator=5.1.0=pyhd8ed1ab_0
---
>   - dbus=1.13.6=h5008d03_3
>   - debugpy=1.5.1=py38h709712a_0
>   - decorator=5.1.1=pyhd8ed1ab_0
97c96
<   - distributed=2021.11.2=py37h89c1867_0
---
>   - distributed=2022.1.0=py38h578d9bd_0
100,101c99
<   - dv3d=8.2.1=pyh9f0ad1d_0
<   - eccodes=2.23.0=h11d1a29_3
---
>   - eccodes=2.24.2=h11d1a29_0
106,108c104,107
<   - esmf=8.2.0=nompi_h61edca3_0
<   - esmpy=8.2.0=nompi_py37hff05f9c_1
<   - expat=2.4.1=h9c3ff4c_0
---
>   - esmf=8.2.0=mpi_mpich_h4975321_100
>   - esmpy=8.2.0=mpi_mpich_py38h9147699_101
>   - executing=0.8.2=pyhd8ed1ab_0
>   - expat=2.4.3=h9c3ff4c_0
111,112c110,111
<   - ffmpeg=4.2.3=h167e202_0
<   - fiona=1.8.20=py37hb7e2723_2
---
>   - findlibs=0.0.2=pyhd8ed1ab_0
>   - fiona=1.8.20=py38hbb147eb_2
115c114,115
<   - folium=0.12.1.post1=pyhd8ed1ab_0
---
>   - flit-core=3.6.0=pyhd8ed1ab_0
>   - folium=0.12.1.post1=pyhd8ed1ab_1
123c123
<   - fonttools=4.28.1=py37h5e8e339_0
---
>   - fonttools=4.28.5=py38h497a2fe_0
127,128c127,128
<   - frozenlist=1.2.0=py37h5e8e339_1
<   - fsspec=2021.11.0=pyhd8ed1ab_0
---
>   - frozenlist=1.2.0=py38h497a2fe_1
>   - fsspec=2022.1.0=pyhd8ed1ab_0
130c130
<   - future=0.18.2=py37h89c1867_4
---
>   - future=0.18.2=py38h578d9bd_4
132,133c132,133
<   - gcsfs=2021.11.0=pyhd8ed1ab_0
<   - gdal=3.3.2=py37hd5a0ba4_2
---
>   - gcsfs=2022.1.0=pyhd8ed1ab_0
>   - gdal=3.3.3=py38hcf2042a_0
135,137c135,136
<   - genutil=8.2.1=py37hb1e94ed_2
<   - geopandas=0.10.2=pyhd8ed1ab_0
<   - geopandas-base=0.10.2=pyha770c72_0
---
>   - geopandas=0.10.2=pyhd8ed1ab_1
>   - geopandas-base=0.10.2=pyha770c72_1
139,141c138,140
<   - geotiff=1.7.0=h4f31c25_0
<   - geoviews=1.9.2=pyhd8ed1ab_0
<   - geoviews-core=1.9.2=pyha770c72_0
---
>   - geotiff=1.7.0=hcfb7246_3
>   - geoviews=1.9.3=pyhd8ed1ab_0
>   - geoviews-core=1.9.3=pyha770c72_0
143d141
<   - ghostscript=9.54.0=h9c3ff4c_1
146,149c144
<   - gitpython=3.1.24=pyhd8ed1ab_0
<   - glib=2.70.1=h780b84a_0
<   - glib-tools=2.70.1=h780b84a_0
<   - gmp=6.2.1=h58526e2_0
---
>   - gitpython=3.1.26=pyhd8ed1ab_0
151c146
<   - google-api-core=2.2.2=pyhd8ed1ab_0
---
>   - google-api-core=2.4.0=pyhd8ed1ab_0
154,156c149,151
<   - google-cloud-core=2.2.1=pyh6c4a22f_0
<   - google-cloud-storage=1.42.3=pyh6c4a22f_1
<   - google-crc32c=1.1.2=py37hab72019_2
---
>   - google-cloud-core=2.2.2=pyh6c4a22f_0
>   - google-cloud-storage=2.0.0=pyh6c4a22f_0
>   - google-crc32c=1.1.2=py38h8838a9a_2
158,164c153,160
<   - googleapis-common-protos=1.53.0=py37h89c1867_1
<   - greenlet=1.1.2=py37hcd2ae1e_1
<   - grpcio=1.42.0=py37hb27c1af_0
<   - gst-plugins-base=1.18.5=hf529b03_2
<   - gstreamer=1.18.5=h9f60fe5_2
<   - h5netcdf=0.11.0=pyhd8ed1ab_0
<   - h5py=3.4.0=nompi_py37hd308b1e_102
---
>   - googleapis-common-protos=1.54.0=py38h578d9bd_0
>   - greenlet=1.1.2=py38h709712a_1
>   - grpcio=1.43.0=py38hdd6454d_0
>   - gst-plugins-base=1.18.5=hf529b03_3
>   - gstreamer=1.18.5=h9f60fe5_3
>   - h11=0.12.0=pyhd8ed1ab_0
>   - h5netcdf=0.13.1=pyhd8ed1ab_0
>   - h5py=3.6.0=nompi_py38hfbb2109_100
167c163
<   - hdf5=1.12.1=nompi_h2750804_102
---
>   - hdf5=1.12.1=mpi_mpich_h9c45103_3
169c165
<   - holoviews=1.14.6=pyhd8ed1ab_0
---
>   - holoviews=1.14.7=pyhd8ed1ab_0
172,176c168,172
<   - idna=2.10=pyh9f0ad1d_0
<   - imagecodecs=2021.8.26=py37hfe5a812_1
<   - imageio=2.9.0=py_0
<   - importlib-metadata=4.8.2=py37h89c1867_0
<   - importlib_metadata=4.8.2=hd8ed1ab_0
---
>   - idna=3.3=pyhd8ed1ab_0
>   - imagecodecs=2021.8.26=py38hb5ce8f7_1
>   - imageio=2.13.5=pyh239f2a4_0
>   - importlib-metadata=4.10.1=py38h578d9bd_0
>   - importlib_metadata=4.10.1=hd8ed1ab_0
179c175
<   - intake=0.6.4=pyhd8ed1ab_0
---
>   - intake=0.6.5=pyhd8ed1ab_0
181c177
<   - intake-geopandas=0.3.0=pyhd8ed1ab_0
---
>   - intake-geopandas=0.4.0=pyhd8ed1ab_0
184,186c180,182
<   - ipykernel=6.5.0=py37h6531663_1
<   - ipyleaflet=0.14.0=pyhd8ed1ab_1
<   - ipython=7.29.0=py37h6531663_2
---
>   - ipykernel=6.7.0=py38he5a9106_0
>   - ipyleaflet=0.15.0=pyhd8ed1ab_3
>   - ipython=8.0.1=py38h578d9bd_0
192c188
<   - jedi=0.18.1=py37h89c1867_0
---
>   - jedi=0.18.1=py38h578d9bd_0
199,201c195,197
<   - jsonpickle=2.0.0=pyhd8ed1ab_0
<   - jsonschema=4.2.1=pyhd8ed1ab_0
<   - jupyter=1.0.0=py37h89c1867_7
---
>   - jsonpickle=2.1.0=pyhd8ed1ab_0
>   - jsonschema=4.4.0=pyhd8ed1ab_0
>   - jupyter=1.0.0=py38h578d9bd_7
206,210c202,206
<   - jupyter_bokeh=3.0.2=pyhd8ed1ab_0
<   - jupyter_client=6.1.12=pyhd8ed1ab_0
<   - jupyter_console=6.4.0=pyhd8ed1ab_1
<   - jupyter_core=4.9.1=py37h89c1867_1
<   - jupyter_server=1.12.0=pyhd8ed1ab_0
---
>   - jupyter_bokeh=3.0.4=pyhd8ed1ab_0
>   - jupyter_client=7.1.1=pyhd8ed1ab_0
>   - jupyter_console=6.4.0=pyhd8ed1ab_0
>   - jupyter_core=4.9.1=py38h578d9bd_1
>   - jupyter_server=1.13.3=pyhd8ed1ab_0
212,215c208,211
<   - jupyterhub=1.5.0=py37h89c1867_1
<   - jupyterhub-base=1.5.0=py37h89c1867_1
<   - jupyterlab=3.2.4=pyhd8ed1ab_0
<   - jupyterlab-git=0.34.0=pyhd8ed1ab_0
---
>   - jupyterhub=2.1.0=hd8ed1ab_0
>   - jupyterhub-base=2.1.0=pyhd8ed1ab_0
>   - jupyterlab=3.2.8=pyhd8ed1ab_0
>   - jupyterlab-git=0.34.2=pyhd8ed1ab_0
219c215
<   - jupyterlab_server=2.8.2=pyhd8ed1ab_0
---
>   - jupyterlab_server=2.10.3=pyhd8ed1ab_0
221c217
<   - jupytext=1.13.1=pyh6002c4b_0
---
>   - jupytext=1.13.6=pyheef035f_0
224c220
<   - kiwisolver=1.3.2=py37h2527ec5_1
---
>   - kiwisolver=1.3.2=py38h1fd1430_1
226,227c222
<   - lame=3.100=h7f98852_1001
<   - lazy-object-proxy=1.6.0=py37h5e8e339_1
---
>   - lazy-object-proxy=1.7.1=py38h497a2fe_0
233c228
<   - libblas=3.9.0=12_linux64_openblas
---
>   - libblas=3.9.0=13_linux64_openblas
237c232
<   - libcblas=3.9.0=12_linux64_openblas
---
>   - libcblas=3.9.0=13_linux64_openblas
239c234
<   - libcf=1.0.3=py37hf9638ec_113
---
>   - libcf=1.0.3=py38h80f51ce_113
242c237
<   - libcurl=7.80.0=h2574ce0_0
---
>   - libcurl=7.81.0=h2574ce0_0
245c240
<   - libdrm=2.4.108=h7f98852_0
---
>   - libdrm=2.4.109=h7f98852_0
253c248
<   - libgdal=3.3.2=hd2dd7ad_2
---
>   - libgdal=3.3.3=h356f897_0
256c251
<   - libglib=2.70.1=h174f98d_0
---
>   - libglib=2.70.2=h174f98d_1
262,263c257
<   - liblapack=3.9.0=12_linux64_openblas
<   - libllvm10=10.0.1=he513fc3_3
---
>   - liblapack=3.9.0=13_linux64_openblas
265,266c259,260
<   - libmicrohttpd=0.9.73=h7f98852_0
<   - libnetcdf=4.8.1=nompi_hb3fd0d9_101
---
>   - libmicrohttpd=0.9.75=h7f98852_0
>   - libnetcdf=4.8.1=mpi_mpich_h319fa22_1
274,275c268,269
<   - libpq=13.5=hd57d9b9_0
<   - libprotobuf=3.19.1=h780b84a_0
---
>   - libpq=13.5=hd57d9b9_1
>   - libprotobuf=3.19.3=h780b84a_0
279c273
<   - libspatialite=5.0.1=h8694cbe_6
---
>   - libspatialite=5.0.1=h8796b1e_9
285c279
<   - libuv=1.41.1=h7f98852_0
---
>   - libuv=1.40.0=h7f98852_0
287c281
<   - libwebp-base=1.2.1=h7f98852_0
---
>   - libwebp-base=1.2.2=h7f98852_0
295c289
<   - llvmlite=0.36.0=py37h9d7f4d0_0
---
>   - llvmlite=0.38.0=py38h4630a5e_0
297c291
<   - lxml=4.6.4=py37h77fd288_0
---
>   - lxml=4.7.1=py38hf1fe3a4_0
305,307c299,301
<   - markupsafe=2.0.1=py37h5e8e339_1
<   - matplotlib=3.5.0=py37h89c1867_0
<   - matplotlib-base=3.5.0=py37h1058ff1_0
---
>   - markupsafe=2.0.1=py38h497a2fe_1
>   - matplotlib=3.5.1=py38h578d9bd_0
>   - matplotlib-base=3.5.1=py38hf4fb855_0
309,312c303,306
<   - mdit-py-plugins=0.2.8=pyhd8ed1ab_0
<   - memory_profiler=0.58.0=py_0
<   - mesalib=21.2.5=h0e4506f_0
<   - mistune=0.8.4=py37h5e8e339_1005
---
>   - mdit-py-plugins=0.3.0=pyhd8ed1ab_0
>   - memory_profiler=0.60.0=pyhd8ed1ab_0
>   - mesalib=21.2.5=h0e4506f_3
>   - mistune=0.8.4=py38h497a2fe_1005
314,316c308,312
<   - more-itertools=8.11.0=pyhd8ed1ab_0
<   - msgpack-python=1.0.2=py37h2527ec5_2
<   - multidict=5.2.0=py37h5e8e339_1
---
>   - mpi=1.0=mpich
>   - mpi4py=3.1.3=py38he865349_0
>   - mpich=3.4.3=h846660c_100
>   - msgpack-python=1.0.3=py38h1fd1430_0
>   - multidict=5.2.0=py38h497a2fe_1
320,324c316,321
<   - mysql-common=8.0.27=ha770c72_1
<   - mysql-libs=8.0.27=hfa10184_1
<   - nbclassic=0.3.4=pyhd8ed1ab_0
<   - nbclient=0.5.9=pyhd8ed1ab_0
<   - nbconvert=6.3.0=py37h89c1867_1
---
>   - mypy_extensions=0.4.3=py38h578d9bd_4
>   - mysql-common=8.0.28=ha770c72_0
>   - mysql-libs=8.0.28=hfa10184_0
>   - nbclassic=0.3.5=pyhd8ed1ab_0
>   - nbclient=0.5.10=pyhd8ed1ab_1
>   - nbconvert=6.4.0=py38h578d9bd_0
330,331c327,328
<   - ncurses=6.2=h58526e2_4
<   - nest-asyncio=1.5.1=pyhd8ed1ab_0
---
>   - ncurses=6.3=h9c3ff4c_0
>   - nest-asyncio=1.5.4=pyhd8ed1ab_0
333,334c330,331
<   - netcdf-fortran=4.5.3=nompi_h2b6e579_106
<   - netcdf4=1.5.8=nompi_py37hf784469_101
---
>   - netcdf-fortran=4.5.4=mpi_mpich_h1364a43_0
>   - netcdf4=1.5.8=nompi_py38h2823cc8_101
337,338c334,335
<   - nodejs=15.14.0=h92b4a50_0
<   - notebook=6.4.6=pyha770c72_0
---
>   - nodejs=16.13.1=hb931c9a_0
>   - notebook=6.4.7=pyha770c72_0
340,343c337,340
<   - nss=3.72=hb5efdd6_0
<   - numba=0.53.1=py37hb11d6e1_1
<   - numcodecs=0.9.1=py37hcd2ae1e_2
<   - numpy=1.21.4=py37h31617e3_0
---
>   - nss=3.74=hb5efdd6_0
>   - numba=0.55.0=py38h4bf6c61_0
>   - numcodecs=0.9.1=py38h709712a_2
>   - numpy=1.21.5=py38h87f13fb_0
347d343
<   - openh264=2.1.1=h780b84a_0
350a347
>   - outcome=1.1.0=pyhd8ed1ab_0
354,355c351,352
<   - pandas=1.3.4=py37he8f5f7f_1
<   - pandoc=2.16.2=h7f98852_0
---
>   - pandas=1.3.5=py38h43a58ef_0
>   - pandoc=2.17.0.1=h7f98852_0
357c354
<   - panel=0.12.4=pyhd8ed1ab_0
---
>   - panel=0.12.6=pyhd8ed1ab_0
359,360c356,357
<   - paramiko=2.8.0=pyhd8ed1ab_0
<   - parso=0.8.2=pyhd8ed1ab_0
---
>   - paramiko=2.9.2=pyhd8ed1ab_0
>   - parso=0.8.3=pyhd8ed1ab_0
361a359
>   - pathspec=0.9.0=pyhd8ed1ab_0
366c364
<   - pillow=8.4.0=py37h0f21c89_0
---
>   - pillow=8.4.0=py38h8e6f84c_0
370,371c368,370
<   - plotly=5.4.0=pyhd8ed1ab_0
<   - pluggy=1.0.0=py37h89c1867_2
---
>   - platformdirs=2.3.0=pyhd8ed1ab_0
>   - plotly=5.5.0=pyhd8ed1ab_0
>   - pluggy=1.0.0=py38h578d9bd_2
375,376c374,375
<   - postgresql=13.5=h2510834_0
<   - proj=8.0.1=h277dcde_0
---
>   - postgresql=13.5=h2510834_1
>   - proj=8.1.1=h277dcde_2
378,379c377,378
<   - prompt-toolkit=3.0.22=pyha770c72_0
<   - prompt_toolkit=3.0.22=hd8ed1ab_0
---
>   - prompt-toolkit=3.0.24=pyha770c72_0
>   - prompt_toolkit=3.0.24=hd8ed1ab_0
381c380
<   - protobuf=3.19.1=py37hcd2ae1e_1
---
>   - protobuf=3.19.3=py38h709712a_0
383c382
<   - psutil=5.8.0=py37h5e8e339_2
---
>   - psutil=5.9.0=py38h497a2fe_0
385a385
>   - pure_eval=0.2.1=pyhd8ed1ab_1
392,393c392,393
<   - pycurl=7.44.1=py37h88a64d2_1
<   - pydantic=1.8.2=py37h5e8e339_2
---
>   - pycurl=7.44.1=py38h996a351_1
>   - pydantic=1.9.0=py38h497a2fe_0
395,398c395,398
<   - pygeos=0.10.2=py37h48c49eb_0
<   - pygments=2.10.0=pyhd8ed1ab_0
<   - pyjwt=2.3.0=pyhd8ed1ab_0
<   - pynacl=1.4.0=py37h5e8e339_3
---
>   - pygeos=0.10.2=py38hb7fe4a8_0
>   - pygments=2.11.2=pyhd8ed1ab_0
>   - pyjwt=2.3.0=pyhd8ed1ab_1
>   - pynacl=1.5.0=py38h497a2fe_0
400,407c400,407
<   - pyparsing=3.0.6=pyhd8ed1ab_0
<   - pyproj=3.2.1=py37h1939ec9_5
<   - pyqt=5.12.3=py37h89c1867_8
<   - pyqt-impl=5.12.3=py37hac37412_8
<   - pyqt5-sip=4.19.18=py37hcd2ae1e_8
<   - pyqtchart=5.12=py37he336c9b_8
<   - pyqtwebengine=5.12.1=py37he336c9b_8
<   - pyrsistent=0.18.0=py37h5e8e339_0
---
>   - pyparsing=3.0.7=pyhd8ed1ab_0
>   - pyproj=3.2.1=py38h80797bf_2
>   - pyqt=5.12.3=py38h578d9bd_8
>   - pyqt-impl=5.12.3=py38h0ffb2e6_8
>   - pyqt5-sip=4.19.18=py38h709712a_8
>   - pyqtchart=5.12=py38h7400c14_8
>   - pyqtwebengine=5.12.1=py38h7400c14_8
>   - pyrsistent=0.18.1=py38h497a2fe_0
409,411c409,411
<   - pysocks=1.7.1=py37h89c1867_4
<   - pytest=6.2.5=py37h89c1867_1
<   - python=3.7.12=hb7a2778_100_cpython
---
>   - pysocks=1.7.1=py38h578d9bd_4
>   - pytest=6.2.5=py38h578d9bd_2
>   - python=3.8.12=hb7a2778_2_cpython
413c413
<   - python-eccodes=2021.03.0=py37hb1e94ed_2
---
>   - python-eccodes=2021.05.1=py38h6c62de6_0
415c415
<   - python_abi=3.7=2_cp37m
---
>   - python_abi=3.8=2_cp38
419,421c419,421
<   - pywavelets=1.2.0=py37hb1e94ed_0
<   - pyyaml=6.0=py37h5e8e339_3
<   - pyzmq=22.3.0=py37h336d617_1
---
>   - pywavelets=1.2.0=py38h6c62de6_1
>   - pyyaml=6.0=py38h497a2fe_3
>   - pyzmq=22.3.0=py38h2035c66_1
423,425c423,426
<   - qtconsole=5.2.0=pyhd8ed1ab_0
<   - qtpy=1.11.2=pyhd8ed1ab_0
<   - rasterio=1.2.6=py37hc20819c_2
---
>   - qtconsole=5.2.2=pyhd8ed1ab_1
>   - qtconsole-base=5.2.2=pyhd8ed1ab_1
>   - qtpy=2.0.0=pyhd8ed1ab_0
>   - rasterio=1.2.10=py38hfd64e68_0
427c428
<   - ravenpy=0.7.5=pyhff6ddc9_0
---
>   - ravenpy=0.7.8=pyh8a188c0_0
430c431
<   - requests=2.25.1=pyhd3deb0d_0
---
>   - requests=2.27.1=pyhd8ed1ab_0
433c434
<   - rioxarray=0.8.0=pyhd8ed1ab_0
---
>   - rioxarray=0.9.1=pyhd8ed1ab_0
435,444c436,445
<   - rsa=4.7.2=pyh44b312d_0
<   - rtree=0.9.7=py37h0b55af0_3
<   - ruamel.yaml=0.17.17=py37h5e8e339_1
<   - ruamel.yaml.clib=0.2.6=py37h5e8e339_0
<   - s3fs=2021.11.0=pyhd8ed1ab_0
<   - scikit-image=0.18.3=py37he8f5f7f_0
<   - scikit-learn=1.0.1=py37hf9e9bfc_2
<   - scipy=1.7.2=py37hf2a6cf1_0
<   - scp=0.14.1=pyhd8ed1ab_0
<   - selenium=3.141.0=py37h5e8e339_1003
---
>   - rsa=4.8=pyhd8ed1ab_0
>   - rtree=0.9.7=py38h02d302b_3
>   - ruamel.yaml=0.17.19=py38h497a2fe_0
>   - ruamel.yaml.clib=0.2.6=py38h497a2fe_0
>   - s3fs=2022.1.0=pyhd8ed1ab_0
>   - scikit-image=0.19.1=py38h43a58ef_0
>   - scikit-learn=1.0.2=py38h1561384_0
>   - scipy=1.7.3=py38h56a6a73_0
>   - scp=0.14.2=pyhd8ed1ab_0
>   - selenium=4.1.0=pyhd8ed1ab_0
446,447c447,448
<   - setuptools=59.2.0=py37h89c1867_0
<   - shapely=1.7.1=py37h48c49eb_5
---
>   - setuptools=59.8.0=py38h578d9bd_0
>   - shapely=1.7.1=py38hb7fe4a8_5
452c453
<   - sniffio=1.2.0=py37h89c1867_2
---
>   - sniffio=1.2.0=py38h578d9bd_2
455,458c456,461
<   - soupsieve=2.3=pyhd8ed1ab_0
<   - sqlalchemy=1.4.27=py37h5e8e339_0
<   - sqlite=3.36.0=h9cd32fc_2
<   - statsmodels=0.13.1=py37hb1e94ed_0
---
>   - soupsieve=2.3.1=pyhd8ed1ab_0
>   - sparse=0.13.0=pyhd8ed1ab_0
>   - sqlalchemy=1.4.31=py38h497a2fe_0
>   - sqlite=3.37.0=h9cd32fc_0
>   - stack_data=0.1.4=pyhd8ed1ab_0
>   - statsmodels=0.13.1=py38h6c62de6_0
461c464
<   - terminado=0.12.1=py37h89c1867_1
---
>   - terminado=0.12.1=py38h578d9bd_1
471c474
<   - tornado=6.1=py37h5e8e339_2
---
>   - tornado=6.1=py38h497a2fe_2
475,476c478,482
<   - typing-extensions=4.0.0=hd8ed1ab_0
<   - typing_extensions=4.0.0=pyha770c72_0
---
>   - trio=0.19.0=py38h578d9bd_1
>   - trio-websocket=0.9.2=pyhd8ed1ab_0
>   - typed-ast=1.5.1=py38h497a2fe_0
>   - typing-extensions=4.0.1=hd8ed1ab_0
>   - typing_extensions=4.0.1=pyha770c72_0
479,484c485,487
<   - udunits2=2.2.27.27=hc3e0081_2
<   - unicodedata2=13.0.0.post2=py37h5e8e339_4
<   - urllib3=1.26.7=pyhd8ed1ab_0
<   - vcs=8.2.1=pyh9f0ad1d_0
<   - voila=0.2.16=pyhd8ed1ab_0
<   - vtk-cdat=8.2.0.8.2=py37_mesalibhead77ed_0
---
>   - unicodedata2=14.0.0=py38h497a2fe_0
>   - urllib3=1.25.11=py_0
>   - voila=0.3.0=pyhd8ed1ab_1
488,489c491,493
<   - websocket-client=1.2.1=py37h89c1867_0
<   - werkzeug=2.0.1=pyhd8ed1ab_0
---
>   - websocket-client=1.2.3=pyhd8ed1ab_0
>   - websockets=10.1=py38h497a2fe_0
>   - werkzeug=2.0.2=pyhd8ed1ab_0
491,496c495,500
<   - wheel=0.37.0=pyhd8ed1ab_1
<   - widgetsnbextension=3.5.2=py37h89c1867_1
<   - wrapt=1.13.3=py37h5e8e339_1
<   - x264=1!152.20180806=h14c3975_0
<   - xarray=0.20.1=pyhd8ed1ab_0
<   - xclim=0.31.0=pyhd8ed1ab_0
---
>   - wheel=0.37.1=pyhd8ed1ab_0
>   - widgetsnbextension=3.5.2=py38h578d9bd_1
>   - wrapt=1.13.3=py38h497a2fe_1
>   - wsproto=1.0.0=py38h578d9bd_2
>   - xarray=0.20.2=pyhd8ed1ab_0
>   - xclim=0.32.1=pyhd8ed1ab_0
498,502c502,506
<   - xesmf=0.6.0=pyhd8ed1ab_0
<   - xeus=2.2.0=hab3612f_0
<   - xeus-python=0.13.5=py37h4b46df4_1
<   - xeus-python-shell=0.1.7=pyhd8ed1ab_0
<   - xhistogram=0.3.0=pyhd8ed1ab_0
---
>   - xesmf=0.6.2=pyhd8ed1ab_0
>   - xeus=2.3.1=hab3612f_0
>   - xeus-python=0.13.6=py38hcf90354_1
>   - xeus-python-shell=0.2.0=pyhd8ed1ab_0
>   - xhistogram=0.3.1=pyhd8ed1ab_0
510c514
<   - xorg-libx11=1.6.12=h36c2ea0_0
---
>   - xorg-libx11=1.7.2=h7f98852_0
512c516
<   - xorg-libxdamage=1.1.5=h7f98852_0
---
>   - xorg-libxdamage=1.1.5=h7f98852_1
514,518c518,522
<   - xorg-libxext=1.3.4=h516909a_0
<   - xorg-libxfixes=5.0.3=h516909a_1004
<   - xorg-libxi=1.7.10=h516909a_0
<   - xorg-libxrandr=1.5.2=h516909a_1
<   - xorg-libxrender=0.9.10=h516909a_1002
---
>   - xorg-libxext=1.3.4=h7f98852_1
>   - xorg-libxfixes=5.0.3=h7f98852_1004
>   - xorg-libxi=1.7.10=h7f98852_0
>   - xorg-libxrandr=1.5.2=h7f98852_1
>   - xorg-libxrender=0.9.10=h7f98852_1003
525c529
<   - xrft=0.4.0=pyhd8ed1ab_0
---
>   - xrft=0.4.1=pyhd8ed1ab_0
527c531
<   - xyzservices=2021.11.0=pyhd8ed1ab_0
---
>   - xyzservices=2022.1.1=pyhd8ed1ab_0
529,530c533,534
<   - yaml=0.2.5=h516909a_0
<   - yarl=1.7.2=py37h5e8e339_1
---
>   - yaml=0.2.5=h7f98852_2
>   - yarl=1.7.2=py38h497a2fe_1
533c537
<   - zfp=0.5.5=h9c3ff4c_7
---
>   - zfp=0.5.5=h9c3ff4c_8
535c539
<   - zipp=3.6.0=pyhd8ed1ab_0
---
>   - zipp=3.7.0=pyhd8ed1ab_0
537c541
<   - zstd=1.5.0=ha95c52a_0
---
>   - zstd=1.5.2=ha95c52a_0
546a551
>     - more-itertools==8.12.0
548c553
<     - pymetalink==6.2
---
>     - pymetalink==6.4
```


- Full new `conda env export`: 

[211221-conda-env-export.yml.txt](https://github.com/Ouranosinc/PAVICS-e2e-workflow-tests/files/7758759/211221-conda-env-export.yml.txt)

[220116.1-conda-env-export.yml.txt](https://github.com/Ouranosinc/PAVICS-e2e-workflow-tests/files/7909175/220116.1-conda-env-export.yml.txt)

[220121-conda-env-export.yml.txt](https://github.com/Ouranosinc/PAVICS-e2e-workflow-tests/files/7922629/220121-conda-env-export.yml.txt)

```
name: birdy
channels:
  - cdat
  - conda-forge
  - defaults
dependencies:
  - _libgcc_mutex=0.1=conda_forge
  - _openmp_mutex=4.5=1_gnu
  - affine=2.3.0=py_0
  - aiobotocore=2.1.0=pyhd8ed1ab_0
  - aiohttp=3.8.1=py38h497a2fe_0
  - aioitertools=0.8.0=pyhd8ed1ab_0
  - aiosignal=1.2.0=pyhd8ed1ab_0
  - alembic=1.7.5=pyhd8ed1ab_0
  - alsa-lib=1.2.3=h516909a_0
  - ansi2html=1.6.0=py38h578d9bd_2
  - anyio=3.5.0=py38h578d9bd_0
  - appdirs=1.4.4=pyh9f0ad1d_0
  - argon2-cffi=21.3.0=pyhd8ed1ab_0
  - argon2-cffi-bindings=21.2.0=py38h497a2fe_1
  - asciitree=0.3.3=py_2
  - asttokens=2.0.5=pyhd8ed1ab_0
  - async-timeout=4.0.2=pyhd8ed1ab_0
  - async_generator=1.10=py_0
  - attrs=21.4.0=pyhd8ed1ab_0
  - babel=2.9.1=pyh44b312d_0
  - backcall=0.2.0=pyh9f0ad1d_0
  - backports=1.0=py_2
  - backports.functools_lru_cache=1.6.4=pyhd8ed1ab_0
  - bcrypt=3.2.0=py38h497a2fe_2
  - beautifulsoup4=4.10.0=pyha770c72_0
  - bias_correction=0.4=pyhd8ed1ab_0
  - birdy=0.8.1=pyh6c4a22f_1
  - black=21.12b0=pyhd8ed1ab_0
  - bleach=4.1.0=pyhd8ed1ab_0
  - blinker=1.4=py_1
  - blosc=1.21.0=h9c3ff4c_0
  - bokeh=2.4.2=py38h578d9bd_0
  - boltons=21.0.0=pyhd8ed1ab_0
  - boost-cpp=1.74.0=h312852a_4
  - botocore=1.23.24=pyhd8ed1ab_0
  - bottleneck=1.3.2=py38h6c62de6_5
  - branca=0.4.2=pyhd8ed1ab_0
  - brotli=1.0.9=h7f98852_6
  - brotli-bin=1.0.9=h7f98852_6
  - brotli-python=1.0.9=py38h709712a_6
  - brotlipy=0.7.0=py38h497a2fe_1003
  - brunsli=0.1=h9c3ff4c_0
  - bzip2=1.0.8=h7f98852_4
  - c-ares=1.18.1=h7f98852_0
  - c-blosc2=2.0.4=h5f21a17_1
  - ca-certificates=2021.10.8=ha878542_0
  - cached-property=1.5.2=hd8ed1ab_1
  - cached_property=1.5.2=pyha770c72_1
  - cachetools=4.2.4=pyhd8ed1ab_0
  - cairo=1.16.0=h6cf1ce9_1008
  - cartopy=0.20.1=py38hf9a4893_1
  - cdat_info=8.2.1=pyhd8ed1ab_2
  - cdms2=3.1.5=py38hc1ef740_14
  - cdtime=3.1.4=py38hc54c017_7
  - certifi=2021.10.8=py38h578d9bd_1
  - certipy=0.1.3=py_0
  - cf_xarray=0.6.3=pyhd8ed1ab_0
  - cffi=1.15.0=py38h3931269_0
  - cfgrib=0.9.9.1=pyhd8ed1ab_1
  - cfitsio=3.470=hb418390_7
  - cftime=1.5.1.1=py38h6c62de6_1
  - charls=2.2.0=h9c3ff4c_0
  - charset-normalizer=2.0.10=pyhd8ed1ab_0
  - click=8.0.3=py38h578d9bd_1
  - click-plugins=1.1.1=py_0
  - cligj=0.7.2=pyhd8ed1ab_1
  - climpred=2.2.0=pyhd8ed1ab_0
  - clisops=0.8.0=pyh6c4a22f_0
  - cloudpickle=2.0.0=pyhd8ed1ab_0
  - colorama=0.4.4=pyh9f0ad1d_0
  - colorcet=3.0.0=pyhd8ed1ab_0
  - configurable-http-proxy=4.5.0=node16_h3b247e2_2
  - coverage=6.2=py38h497a2fe_0
  - cryptography=36.0.1=py38h3e25421_0
  - curl=7.81.0=h2574ce0_0
  - cycler=0.11.0=pyhd8ed1ab_0
  - cytoolz=0.11.2=py38h497a2fe_1
  - dash=2.0.0=pyhd8ed1ab_0
  - dask=2022.1.0=pyhd8ed1ab_0
  - dask-core=2022.1.0=pyhd8ed1ab_0
  - dataclasses=0.8=pyhc8e2a94_3
  - datashader=0.13.0=pyh6c4a22f_0
  - datashape=0.5.4=py_1
  - dbus=1.13.6=h5008d03_3
  - debugpy=1.5.1=py38h709712a_0
  - decorator=5.1.1=pyhd8ed1ab_0
  - defusedxml=0.7.1=pyhd8ed1ab_0
  - descartes=1.1.0=py_4
  - distarray=2.12.2=pyhd8ed1ab_2
  - distributed=2022.1.0=py38h578d9bd_0
  - docopt=0.6.2=py_1
  - docrep=0.3.2=pyh44b312d_0
  - eccodes=2.24.2=h11d1a29_0
  - elfutils=0.186=he364ef2_0
  - entrypoints=0.3=pyhd8ed1ab_1003
  - eofs=1.4.0=py_0
  - esgf-compute-api=2.3.9=0_h1234567_
  - esmf=8.2.0=mpi_mpich_h4975321_100
  - esmpy=8.2.0=mpi_mpich_py38h9147699_101
  - executing=0.8.2=pyhd8ed1ab_0
  - expat=2.4.3=h9c3ff4c_0
  - fasteners=0.16=pyhd8ed1ab_0
  - fastprogress=1.0.0=py_0
  - findlibs=0.0.2=pyhd8ed1ab_0
  - fiona=1.8.20=py38hbb147eb_2
  - flask=2.0.2=pyhd8ed1ab_0
  - flask-compress=1.10.1=pyhd8ed1ab_0
  - flit-core=3.6.0=pyhd8ed1ab_0
  - folium=0.12.1.post1=pyhd8ed1ab_1
  - font-ttf-dejavu-sans-mono=2.37=hab24e00_0
  - font-ttf-inconsolata=3.000=h77eed37_0
  - font-ttf-source-code-pro=2.038=h77eed37_0
  - font-ttf-ubuntu=0.83=hab24e00_0
  - fontconfig=2.13.1=hba837de_1005
  - fonts-conda-ecosystem=1=0
  - fonts-conda-forge=1=0
  - fonttools=4.28.5=py38h497a2fe_0
  - freeglut=3.2.1=h9c3ff4c_2
  - freetype=2.10.4=h0708190_1
  - freexl=1.0.6=h7f98852_0
  - frozenlist=1.2.0=py38h497a2fe_1
  - fsspec=2022.1.0=pyhd8ed1ab_0
  - funcsigs=1.0.2=py_3
  - future=0.18.2=py38h578d9bd_4
  - g2clib=1.6.3=heb9ad7a_1
  - gcsfs=2022.1.0=pyhd8ed1ab_0
  - gdal=3.3.3=py38hcf2042a_0
  - geckodriver=0.30.0=h3146498_0
  - geopandas=0.10.2=pyhd8ed1ab_1
  - geopandas-base=0.10.2=pyha770c72_1
  - geos=3.9.1=h9c3ff4c_2
  - geotiff=1.7.0=hcfb7246_3
  - geoviews=1.9.3=pyhd8ed1ab_0
  - geoviews-core=1.9.3=pyha770c72_0
  - gettext=0.19.8.1=h73d1719_1008
  - giflib=5.2.1=h36c2ea0_2
  - gitdb=4.0.9=pyhd8ed1ab_0
  - gitpython=3.1.26=pyhd8ed1ab_0
  - gnutls=3.6.13=h85f3911_1
  - google-api-core=2.4.0=pyhd8ed1ab_0
  - google-auth=2.3.3=pyh6c4a22f_0
  - google-auth-oauthlib=0.4.6=pyhd8ed1ab_0
  - google-cloud-core=2.2.2=pyh6c4a22f_0
  - google-cloud-storage=2.0.0=pyh6c4a22f_0
  - google-crc32c=1.1.2=py38h8838a9a_2
  - google-resumable-media=2.1.0=pyh6c4a22f_0
  - googleapis-common-protos=1.54.0=py38h578d9bd_0
  - greenlet=1.1.2=py38h709712a_1
  - grpcio=1.43.0=py38hdd6454d_0
  - gst-plugins-base=1.18.5=hf529b03_3
  - gstreamer=1.18.5=h9f60fe5_3
  - h11=0.12.0=pyhd8ed1ab_0
  - h5netcdf=0.13.1=pyhd8ed1ab_0
  - h5py=3.6.0=nompi_py38hfbb2109_100
  - haversine=2.5.1=pyhd8ed1ab_0
  - hdf4=4.2.15=h10796ff_3
  - hdf5=1.12.1=mpi_mpich_h9c45103_3
  - heapdict=1.0.1=py_0
  - holoviews=1.14.7=pyhd8ed1ab_0
  - hvplot=0.7.3=pyh6c4a22f_0
  - icu=68.2=h9c3ff4c_0
  - idna=3.3=pyhd8ed1ab_0
  - imagecodecs=2021.8.26=py38hb5ce8f7_1
  - imageio=2.13.5=pyh239f2a4_0
  - importlib-metadata=4.10.1=py38h578d9bd_0
  - importlib_metadata=4.10.1=hd8ed1ab_0
  - importlib_resources=5.4.0=pyhd8ed1ab_0
  - iniconfig=1.1.1=pyh9f0ad1d_0
  - intake=0.6.5=pyhd8ed1ab_0
  - intake-esm=2021.8.17=pyhd8ed1ab_0
  - intake-geopandas=0.4.0=pyhd8ed1ab_0
  - intake-thredds=2021.6.16=pyhd8ed1ab_0
  - intake-xarray=0.5.0=pyhd8ed1ab_0
  - ipykernel=6.7.0=py38he5a9106_0
  - ipyleaflet=0.15.0=pyhd8ed1ab_3
  - ipython=8.0.1=py38h578d9bd_0
  - ipython_genutils=0.2.0=py_1
  - ipywidgets=7.6.5=pyhd8ed1ab_0
  - itsdangerous=2.0.1=pyhd8ed1ab_0
  - jasper=2.0.33=ha77e612_0
  - jbig=2.1=h7f98852_2003
  - jedi=0.18.1=py38h578d9bd_0
  - jinja2=3.0.3=pyhd8ed1ab_0
  - jmespath=0.10.0=pyh9f0ad1d_0
  - joblib=1.1.0=pyhd8ed1ab_0
  - jpeg=9d=h36c2ea0_0
  - json-c=0.15=h98cffda_0
  - json5=0.9.5=pyh9f0ad1d_0
  - jsonpickle=2.1.0=pyhd8ed1ab_0
  - jsonschema=4.4.0=pyhd8ed1ab_0
  - jupyter=1.0.0=py38h578d9bd_7
  - jupyter-archive=3.2.1=pyhd8ed1ab_0
  - jupyter-dash=0.4.0=pyhd8ed1ab_0
  - jupyter-resource-usage=0.5.1=pyhd8ed1ab_0
  - jupyter-server-mathjax=0.2.3=pyhd8ed1ab_0
  - jupyter_bokeh=3.0.4=pyhd8ed1ab_0
  - jupyter_client=7.1.1=pyhd8ed1ab_0
  - jupyter_console=6.4.0=pyhd8ed1ab_0
  - jupyter_core=4.9.1=py38h578d9bd_1
  - jupyter_server=1.13.3=pyhd8ed1ab_0
  - jupyter_telemetry=0.1.0=pyhd8ed1ab_1
  - jupyterhub=2.1.0=hd8ed1ab_0
  - jupyterhub-base=2.1.0=pyhd8ed1ab_0
  - jupyterlab=3.2.8=pyhd8ed1ab_0
  - jupyterlab-git=0.34.2=pyhd8ed1ab_0
  - jupyterlab-system-monitor=0.8.0=pyhd8ed1ab_1
  - jupyterlab-topbar=0.6.1=pyhd8ed1ab_2
  - jupyterlab_pygments=0.1.2=pyh9f0ad1d_0
  - jupyterlab_server=2.10.3=pyhd8ed1ab_0
  - jupyterlab_widgets=1.0.2=pyhd8ed1ab_0
  - jupytext=1.13.6=pyheef035f_0
  - jxrlib=1.1=h7f98852_2
  - kealib=1.4.14=h87e4c3c_3
  - kiwisolver=1.3.2=py38h1fd1430_1
  - krb5=1.19.2=hcc1bbae_3
  - lazy-object-proxy=1.7.1=py38h497a2fe_0
  - lcms2=2.12=hddcbb42_0
  - ld_impl_linux-64=2.36.1=hea4e1c9_2
  - lerc=3.0=h9c3ff4c_0
  - libaec=1.0.6=h9c3ff4c_0
  - libarchive=3.5.2=hccf745f_1
  - libblas=3.9.0=13_linux64_openblas
  - libbrotlicommon=1.0.9=h7f98852_6
  - libbrotlidec=1.0.9=h7f98852_6
  - libbrotlienc=1.0.9=h7f98852_6
  - libcblas=3.9.0=13_linux64_openblas
  - libcdms=3.1.2=h3bdf4ba_117
  - libcf=1.0.3=py38h80f51ce_113
  - libclang=11.1.0=default_ha53f305_1
  - libcrc32c=1.1.2=h9c3ff4c_0
  - libcurl=7.81.0=h2574ce0_0
  - libdap4=3.20.6=hd7c4107_2
  - libdeflate=1.8=h7f98852_0
  - libdrm=2.4.109=h7f98852_0
  - libdrs=3.1.2=hf593df3_118
  - libdrs_f=3.1.2=h7e76ec7_114
  - libedit=3.1.20191231=he28a2e2_2
  - libev=4.33=h516909a_1
  - libevent=2.1.10=h9b69904_4
  - libffi=3.4.2=h7f98852_5
  - libgcc-ng=11.2.0=h1d223b6_11
  - libgdal=3.3.3=h356f897_0
  - libgfortran-ng=11.2.0=h69a702a_11
  - libgfortran5=11.2.0=h5c6108e_11
  - libglib=2.70.2=h174f98d_1
  - libglu=9.0.0=he1b5a44_1001
  - libgomp=11.2.0=h1d223b6_11
  - libiconv=1.16=h516909a_0
  - libidn2=2.3.2=h7f98852_0
  - libkml=1.3.0=h238a007_1014
  - liblapack=3.9.0=13_linux64_openblas
  - libllvm11=11.1.0=hf817b99_2
  - libmicrohttpd=0.9.75=h7f98852_0
  - libnetcdf=4.8.1=mpi_mpich_h319fa22_1
  - libnghttp2=1.43.0=h812cca2_1
  - libnsl=2.0.0=h7f98852_0
  - libogg=1.3.4=h7f98852_1
  - libopenblas=0.3.18=pthreads_h8fe5266_0
  - libopus=1.3.1=h7f98852_1
  - libpciaccess=0.16=h516909a_0
  - libpng=1.6.37=h21135ba_2
  - libpq=13.5=hd57d9b9_1
  - libprotobuf=3.19.3=h780b84a_0
  - librttopo=1.1.0=h1185371_6
  - libsodium=1.0.18=h36c2ea0_1
  - libspatialindex=1.9.3=h9c3ff4c_4
  - libspatialite=5.0.1=h8796b1e_9
  - libssh2=1.10.0=ha56f1ee_2
  - libstdcxx-ng=11.2.0=he4da1e4_11
  - libtiff=4.3.0=h6f004c6_2
  - libunistring=0.9.10=h7f98852_0
  - libuuid=2.32.1=h7f98852_1000
  - libuv=1.40.0=h7f98852_0
  - libvorbis=1.3.7=h9c3ff4c_0
  - libwebp-base=1.2.2=h7f98852_0
  - libxcb=1.13=h7f98852_1004
  - libxkbcommon=1.0.3=he3ba5ed_0
  - libxml2=2.9.12=h72842e0_0
  - libxslt=1.1.33=h15afd5d_2
  - libzip=1.8.0=h4de3113_1
  - libzlib=1.2.11=h36c2ea0_1013
  - libzopfli=1.0.3=h9c3ff4c_0
  - llvmlite=0.38.0=py38h4630a5e_0
  - locket=0.2.0=py_2
  - lxml=4.7.1=py38hf1fe3a4_0
  - lz4-c=1.9.3=h9c3ff4c_1
  - lzo=2.10=h516909a_1000
  - mako=1.1.6=pyhd8ed1ab_0
  - mamba_gator=5.1.2=pyhd8ed1ab_0
  - mapclassify=2.4.3=pyhd8ed1ab_0
  - markdown=3.3.6=pyhd8ed1ab_0
  - markdown-it-py=1.1.0=pyhd8ed1ab_0
  - markupsafe=2.0.1=py38h497a2fe_1
  - matplotlib=3.5.1=py38h578d9bd_0
  - matplotlib-base=3.5.1=py38hf4fb855_0
  - matplotlib-inline=0.1.3=pyhd8ed1ab_0
  - mdit-py-plugins=0.3.0=pyhd8ed1ab_0
  - memory_profiler=0.60.0=pyhd8ed1ab_0
  - mesalib=21.2.5=h0e4506f_3
  - mistune=0.8.4=py38h497a2fe_1005
  - monotonic=1.5=py_0
  - mpi=1.0=mpich
  - mpi4py=3.1.3=py38he865349_0
  - mpich=3.4.3=h846660c_100
  - msgpack-python=1.0.3=py38h1fd1430_0
  - multidict=5.2.0=py38h497a2fe_1
  - multipledispatch=0.6.0=py_0
  - munch=2.5.0=py_0
  - munkres=1.1.4=pyh9f0ad1d_0
  - mypy_extensions=0.4.3=py38h578d9bd_4
  - mysql-common=8.0.28=ha770c72_0
  - mysql-libs=8.0.28=hfa10184_0
  - nbclassic=0.3.5=pyhd8ed1ab_0
  - nbclient=0.5.10=pyhd8ed1ab_1
  - nbconvert=6.4.0=py38h578d9bd_0
  - nbdime=3.1.1=pyhd8ed1ab_0
  - nbformat=5.1.3=pyhd8ed1ab_0
  - nbresuse=0.4.0=pyhd8ed1ab_0
  - nbval=0.9.6=pyh9f0ad1d_0
  - nc-time-axis=1.4.0=pyhd8ed1ab_0
  - ncurses=6.3=h9c3ff4c_0
  - nest-asyncio=1.5.4=pyhd8ed1ab_0
  - nested_dict=1.61=pyhd3deb0d_0
  - netcdf-fortran=4.5.4=mpi_mpich_h1364a43_0
  - netcdf4=1.5.8=nompi_py38h2823cc8_101
  - nettle=3.6=he412f7d_0
  - networkx=2.6.3=pyhd8ed1ab_1
  - nodejs=16.13.1=hb931c9a_0
  - notebook=6.4.7=pyha770c72_0
  - nspr=4.32=h9c3ff4c_1
  - nss=3.74=hb5efdd6_0
  - numba=0.55.0=py38h4bf6c61_0
  - numcodecs=0.9.1=py38h709712a_2
  - numpy=1.21.5=py38h87f13fb_0
  - oauthlib=3.1.1=pyhd8ed1ab_0
  - olefile=0.46=pyh9f0ad1d_1
  - openblas=0.3.18=pthreads_h4748800_0
  - openjpeg=2.4.0=hb52868f_1
  - openssl=1.1.1l=h7f98852_0
  - ostrich=21.03.16=h4bd325d_1
  - outcome=1.1.0=pyhd8ed1ab_0
  - owslib=0.25.0=pyhd8ed1ab_0
  - packaging=21.3=pyhd8ed1ab_0
  - pamela=1.0.0=py_0
  - pandas=1.3.5=py38h43a58ef_0
  - pandoc=2.17.0.1=h7f98852_0
  - pandocfilters=1.5.0=pyhd8ed1ab_0
  - panel=0.12.6=pyhd8ed1ab_0
  - param=1.12.0=pyh6c4a22f_0
  - paramiko=2.9.2=pyhd8ed1ab_0
  - parso=0.8.3=pyhd8ed1ab_0
  - partd=1.2.0=pyhd8ed1ab_0
  - pathspec=0.9.0=pyhd8ed1ab_0
  - patsy=0.5.2=pyhd8ed1ab_0
  - pcre=8.45=h9c3ff4c_0
  - pexpect=4.8.0=pyh9f0ad1d_2
  - pickleshare=0.7.5=py_1003
  - pillow=8.4.0=py38h8e6f84c_0
  - pint=0.18=pyhd8ed1ab_0
  - pip=21.3.1=pyhd8ed1ab_0
  - pixman=0.40.0=h36c2ea0_0
  - platformdirs=2.3.0=pyhd8ed1ab_0
  - plotly=5.5.0=pyhd8ed1ab_0
  - pluggy=1.0.0=py38h578d9bd_2
  - pooch=1.5.2=pyhd8ed1ab_0
  - poppler=21.09.0=ha39eefc_3
  - poppler-data=0.4.11=hd8ed1ab_0
  - postgresql=13.5=h2510834_1
  - proj=8.1.1=h277dcde_2
  - prometheus_client=0.12.0=pyhd8ed1ab_0
  - prompt-toolkit=3.0.24=pyha770c72_0
  - prompt_toolkit=3.0.24=hd8ed1ab_0
  - properscoring=0.1=py_0
  - protobuf=3.19.3=py38h709712a_0
  - pscript=0.7.5=pyhd3deb0d_0
  - psutil=5.9.0=py38h497a2fe_0
  - pthread-stubs=0.4=h36c2ea0_1001
  - ptyprocess=0.7.0=pyhd3deb0d_0
  - pure_eval=0.2.1=pyhd8ed1ab_1
  - py=1.11.0=pyh6c4a22f_0
  - pyasn1=0.4.8=py_0
  - pyasn1-modules=0.2.7=py_0
  - pycparser=2.21=pyhd8ed1ab_0
  - pyct=0.4.6=py_0
  - pyct-core=0.4.6=py_0
  - pycurl=7.44.1=py38h996a351_1
  - pydantic=1.9.0=py38h497a2fe_0
  - pydap=3.2.2=pyh9f0ad1d_1001
  - pygeos=0.10.2=py38hb7fe4a8_0
  - pygments=2.11.2=pyhd8ed1ab_0
  - pyjwt=2.3.0=pyhd8ed1ab_1
  - pynacl=1.5.0=py38h497a2fe_0
  - pyopenssl=21.0.0=pyhd8ed1ab_0
  - pyparsing=3.0.7=pyhd8ed1ab_0
  - pyproj=3.2.1=py38h80797bf_2
  - pyqt=5.12.3=py38h578d9bd_8
  - pyqt-impl=5.12.3=py38h0ffb2e6_8
  - pyqt5-sip=4.19.18=py38h709712a_8
  - pyqtchart=5.12=py38h7400c14_8
  - pyqtwebengine=5.12.1=py38h7400c14_8
  - pyrsistent=0.18.1=py38h497a2fe_0
  - pyshp=2.1.3=pyh44b312d_0
  - pysocks=1.7.1=py38h578d9bd_4
  - pytest=6.2.5=py38h578d9bd_2
  - python=3.8.12=hb7a2778_2_cpython
  - python-dateutil=2.8.2=pyhd8ed1ab_0
  - python-eccodes=2021.05.1=py38h6c62de6_0
  - python-json-logger=2.0.1=pyh9f0ad1d_0
  - python_abi=3.8=2_cp38
  - pytz=2021.3=pyhd8ed1ab_0
  - pyu2f=0.1.5=pyhd8ed1ab_0
  - pyviz_comms=2.1.0=pyhd8ed1ab_0
  - pywavelets=1.2.0=py38h6c62de6_1
  - pyyaml=6.0=py38h497a2fe_3
  - pyzmq=22.3.0=py38h2035c66_1
  - qt=5.12.9=hda022c4_4
  - qtconsole=5.2.2=pyhd8ed1ab_1
  - qtconsole-base=5.2.2=pyhd8ed1ab_1
  - qtpy=2.0.0=pyhd8ed1ab_0
  - rasterio=1.2.10=py38hfd64e68_0
  - raven-hydro=3.0.4.322=ha46ed4c_2
  - ravenpy=0.7.8=pyh8a188c0_0
  - readline=8.1=h46c0cb4_0
  - regionmask=0.8.0=pyhd8ed1ab_1
  - requests=2.27.1=pyhd8ed1ab_0
  - requests-oauthlib=1.3.0=pyh9f0ad1d_0
  - retrying=1.3.3=py_2
  - rioxarray=0.9.1=pyhd8ed1ab_0
  - roocs-utils=0.5.0=pyh6c4a22f_0
  - rsa=4.8=pyhd8ed1ab_0
  - rtree=0.9.7=py38h02d302b_3
  - ruamel.yaml=0.17.19=py38h497a2fe_0
  - ruamel.yaml.clib=0.2.6=py38h497a2fe_0
  - s3fs=2022.1.0=pyhd8ed1ab_0
  - scikit-image=0.19.1=py38h43a58ef_0
  - scikit-learn=1.0.2=py38h1561384_0
  - scipy=1.7.3=py38h56a6a73_0
  - scp=0.14.2=pyhd8ed1ab_0
  - selenium=4.1.0=pyhd8ed1ab_0
  - send2trash=1.8.0=pyhd8ed1ab_0
  - setuptools=59.8.0=py38h578d9bd_0
  - shapely=1.7.1=py38hb7fe4a8_5
  - siphon=0.9=pyhd8ed1ab_2
  - six=1.16.0=pyh6c4a22f_0
  - smmap=3.0.5=pyh44b312d_0
  - snappy=1.1.8=he1b5a44_3
  - sniffio=1.2.0=py38h578d9bd_2
  - snuggs=1.4.7=py_0
  - sortedcontainers=2.4.0=pyhd8ed1ab_0
  - soupsieve=2.3.1=pyhd8ed1ab_0
  - sparse=0.13.0=pyhd8ed1ab_0
  - sqlalchemy=1.4.31=py38h497a2fe_0
  - sqlite=3.37.0=h9cd32fc_0
  - stack_data=0.1.4=pyhd8ed1ab_0
  - statsmodels=0.13.1=py38h6c62de6_0
  - tblib=1.7.0=pyhd8ed1ab_0
  - tenacity=8.0.1=pyhd8ed1ab_0
  - terminado=0.12.1=py38h578d9bd_1
  - testpath=0.5.0=pyhd8ed1ab_0
  - threadpoolctl=3.0.0=pyh8a188c0_0
  - threddsclient=0.4.2=py_0
  - tifffile=2021.11.2=pyhd8ed1ab_0
  - tiledb=2.3.4=he87e0bf_0
  - tk=8.6.11=h27826a3_1
  - toml=0.10.2=pyhd8ed1ab_0
  - tomli=1.2.2=pyhd8ed1ab_0
  - toolz=0.11.2=pyhd8ed1ab_0
  - tornado=6.1=py38h497a2fe_2
  - tqdm=4.62.3=pyhd8ed1ab_0
  - traitlets=5.1.1=pyhd8ed1ab_0
  - traittypes=0.2.1=pyh9f0ad1d_2
  - trio=0.19.0=py38h578d9bd_1
  - trio-websocket=0.9.2=pyhd8ed1ab_0
  - typed-ast=1.5.1=py38h497a2fe_0
  - typing-extensions=4.0.1=hd8ed1ab_0
  - typing_extensions=4.0.1=pyha770c72_0
  - tzcode=2021e=h7f98852_0
  - tzdata=2021e=he74cb21_0
  - unicodedata2=14.0.0=py38h497a2fe_0
  - urllib3=1.25.11=py_0
  - voila=0.3.0=pyhd8ed1ab_1
  - wcwidth=0.2.5=pyh9f0ad1d_2
  - webencodings=0.5.1=py_1
  - webob=1.8.7=pyhd8ed1ab_0
  - websocket-client=1.2.3=pyhd8ed1ab_0
  - websockets=10.1=py38h497a2fe_0
  - werkzeug=2.0.2=pyhd8ed1ab_0
  - wget=1.20.3=ha56f1ee_1
  - wheel=0.37.1=pyhd8ed1ab_0
  - widgetsnbextension=3.5.2=py38h578d9bd_1
  - wrapt=1.13.3=py38h497a2fe_1
  - wsproto=1.0.0=py38h578d9bd_2
  - xarray=0.20.2=pyhd8ed1ab_0
  - xclim=0.32.1=pyhd8ed1ab_0
  - xerces-c=3.2.3=h9d8b166_3
  - xesmf=0.6.2=pyhd8ed1ab_0
  - xeus=2.3.1=hab3612f_0
  - xeus-python=0.13.6=py38hcf90354_1
  - xeus-python-shell=0.2.0=pyhd8ed1ab_0
  - xhistogram=0.3.1=pyhd8ed1ab_0
  - xorg-damageproto=1.2.1=h7f98852_1002
  - xorg-fixesproto=5.0=h7f98852_1002
  - xorg-glproto=1.4.17=h7f98852_1002
  - xorg-inputproto=2.3.2=h7f98852_1002
  - xorg-kbproto=1.0.7=h7f98852_1002
  - xorg-libice=1.0.10=h7f98852_0
  - xorg-libsm=1.2.3=hd9c2040_1000
  - xorg-libx11=1.7.2=h7f98852_0
  - xorg-libxau=1.0.9=h7f98852_0
  - xorg-libxdamage=1.1.5=h7f98852_1
  - xorg-libxdmcp=1.1.3=h7f98852_0
  - xorg-libxext=1.3.4=h7f98852_1
  - xorg-libxfixes=5.0.3=h7f98852_1004
  - xorg-libxi=1.7.10=h7f98852_0
  - xorg-libxrandr=1.5.2=h7f98852_1
  - xorg-libxrender=0.9.10=h7f98852_1003
  - xorg-randrproto=1.5.0=h7f98852_1001
  - xorg-renderproto=0.11.1=h7f98852_1002
  - xorg-util-macros=1.19.3=h7f98852_0
  - xorg-xextproto=7.3.0=h7f98852_1002
  - xorg-xf86vidmodeproto=2.3.1=h7f98852_1002
  - xorg-xproto=7.0.31=h7f98852_1007
  - xrft=0.4.1=pyhd8ed1ab_0
  - xskillscore=0.0.24=pyhd8ed1ab_0
  - xyzservices=2022.1.1=pyhd8ed1ab_0
  - xz=5.2.5=h516909a_1
  - yaml=0.2.5=h7f98852_2
  - yarl=1.7.2=py38h497a2fe_1
  - zarr=2.10.3=pyhd8ed1ab_0
  - zeromq=4.3.4=h9c3ff4c_1
  - zfp=0.5.5=h9c3ff4c_8
  - zict=2.0.0=py_0
  - zipp=3.7.0=pyhd8ed1ab_0
  - zlib=1.2.11=h36c2ea0_1013
  - zstd=1.5.2=ha95c52a_0
  - pip:
    - astunparse==1.6.3
    - colour==0.1.5
    - geojson==2.5.0
    - handcalcs==1.4.1
    - innerscope==0.5.0
    - ipython-blocking==0.3.1
    - jupyterlab-logout==0.5.0
    - jupyternotify==0.1.15
    - more-itertools==8.12.0
    - pixiedust==1.1.19
    - pymetalink==6.4
    - pytest-tornasync==0.6.0.post2
    - requests-magpie==0.1.1
prefix: /opt/conda/envs/birdy
```
  • Loading branch information
@tlvu
tlvu committed Jan 27, 2022
2 parents 8b18c96 + 34d38cc commit 607ec58
Show file tree
Hide file tree
Showing 8 changed files with 131 additions and 58 deletions.
2 changes: 1 addition & 1 deletion Jenkinsfile
View file
Expand Up @@ -10,7 +10,7 @@ pipeline {
// https://jenkins.io/doc/book/pipeline/syntax/
agent {
docker {
image "pavics/workflow-tests:211123-update211216"
image "pavics/workflow-tests:220121"
label 'linux && docker'
}
}
Expand Down
2 changes: 1 addition & 1 deletion binder/Dockerfile
View file
@@ -1,4 +1,4 @@
FROM pavics/workflow-tests:211123-update211216
FROM pavics/workflow-tests:220121

USER root

Expand Down
3 changes: 2 additions & 1 deletion conftest.py
View file
Expand Up @@ -2,4 +2,5 @@ def pytest_collectstart(collector):
# Make sure ancestor folder name do not end with `.ipynb`, else we have
# AttributeError: 'Session' object has no attribute 'skip_compare'.
if collector.fspath and collector.fspath.ext == '.ipynb':
collector.skip_compare += 'text/html', 'application/javascript',
collector.skip_compare += ('text/html', 'application/javascript',
'application/vnd.holoviews_load.v0+json',)
80 changes: 56 additions & 24 deletions docker/Dockerfile
View file
@@ -1,6 +1,10 @@
FROM continuumio/miniconda3

RUN conda update conda
# Use mamba for much improved performance over conda.
# The 'channel_priority strict' did help conda but it was not enough.
RUN conda update conda -n base && \
conda install mamba -n base -c defaults -c conda-forge && \
conda config --set channel_priority strict

# to checkout other notebooks and to run pip install
RUN apt-get update && \
Expand All @@ -9,18 +13,40 @@ RUN apt-get update && \
firefox-esr x11-utils && \
apt-get clean

COPY environment.yml /environment.yml

# needed for our specific jenkins
# Create user jenkins for our Jenkins e2e notebooks test suite.
# Change /opt/conda folder permissions for jupyter-conda extension.
RUN groupadd --gid 1000 jenkins \
&& useradd --uid 1000 --gid jenkins --create-home jenkins
&& useradd --uid 1000 --gid jenkins --create-home jenkins && \
chmod -R a+rwx /opt/conda

# Change these folders' permissions for jupyter-conda extension
RUN chmod -R a+rwx /opt/conda
COPY environment.yml /environment.yml

# create env "birdy"
# use umask 0000 so that the files for the new environment are usable by user 'jenkins' for the jupyter-conda-extension
RUN umask 0000 && conda env create -f /environment.yml
#
# Perform 2 stages install because one single 'conda env create -f
# /environment.yml' was taking forever to complete, same with mamba.
# Had to do this 2 stages install. 2 stages install was also taking forever
# with conda so had to switch to mamba.
#
# One single 'conda env create -f /environment.yml' takes forever because we
# removed all direct dependencies of xclim and ravenpy in /environment.yml for
# dependencies pinning by xclim and ravenpy to take effect. This results in
# conda having a lot more packages to "solve" and it seems the solver
# performance dropped exponentially with the number of packages to solve.
#
# Conda was stuck at this step:
# DEBUG conda.common._logic:_run_sat(607): Invoking SAT with clause count: 2500273
#
# Pin python=3.9 because python 3.10 cause this error:
# Encountered problems while solving:
# - package cartopy-0.20.1-py310h902574e_5 requires geos >=3.10.1,<3.10.2.0a0, but none of the providers can be installed
# This means there is no py310 build for geos package but https://anaconda.org/conda-forge/geos/files seems to be python indepdendent !
# Pin python=3.8 because according to DavidH, xESMF has not been tested with 3.9 yet.
RUN umask 0000 && \
mamba create --name birdy --channel conda-forge --channel defaults xclim ravenpy python=3.8 && \
mamba env update --name birdy --file /environment.yml && \
conda remove mamba -n base

# alternate way to 'source activate birdy'
ENV PATH="/opt/conda/envs/birdy/bin:$PATH"
Expand All @@ -33,35 +59,41 @@ RUN python -m ipykernel install --name birdy
# anything accidentally
# this is for debug only, all dependencies should be specified in
# environment.yml above
# RUN conda install -c conda-forge -c cdat -c bokeh -c plotly -c defaults -n birdy nbdime
# RUN mamba install -c conda-forge -c cdat -c bokeh -c plotly -c defaults -n birdy nbdime

# build jupyterlab extensions installed by conda, see `jupyter labextension list`
# Supposedly not needed with jupyterlab v3 anymore but see
# https://github.com/jupyterlab/jupyterlab/issues/11726#issuecomment-998901247
# TODO: remove 'jupyter lab build' step once all extensions move to prebuilt extensions,
# see comment https://github.com/jupyterlab/jupyterlab/issues/11726#issuecomment-998917305
# Currently jupyter-dash is holding back this step, see
# https://github.com/plotly/jupyter-dash/issues/49
RUN jupyter lab build

# for ipywidgets to work with jupyter lab (notebooks works out of the box)
RUN jupyter labextension install @jupyter-widgets/jupyterlab-manager \
&& jupyter serverextension enable voila --sys-prefix \
&& jupyter labextension install @jupyter-widgets/jupyterlab-manager jupyter-leaflet \
&& jupyter labextension install jupyterlab-topbar-text \
jupyterlab-theme-toggle
RUN jupyter serverextension enable voila --sys-prefix
# && jupyter labextension install jupyterlab-clipboard

ADD https://raw.githubusercontent.com/jupyter/docker-stacks/master/base-notebook/start.sh /usr/local/bin/
ADD https://raw.githubusercontent.com/jupyter/docker-stacks/master/base-notebook/start-singleuser.sh /usr/local/bin/
ADD https://raw.githubusercontent.com/jupyter/docker-stacks/master/base-notebook/start-notebook.sh /usr/local/bin/
ADD https://raw.githubusercontent.com/jupyter/docker-stacks/master/base-notebook/fix-permissions /usr/local/bin/
ADD https://raw.githubusercontent.com/jupyter/docker-stacks/master/base-notebook/jupyter_notebook_config.py /etc/jupyter/
RUN chmod a+rx /usr/local/bin/start.sh /usr/local/bin/start-singleuser.sh /usr/local/bin/start-notebook.sh /usr/local/bin/fix-permissions; \
chmod a+r /etc/jupyter/jupyter_notebook_config.py
# This should be "master" but commit
# https://github.com/jupyter/docker-stacks/commit/c772e98ac794173d6ed83a08ec249038b27ca3be
# is breaking with us since we do not have user jovyan.
ENV DOCKER_STACKS_COMMIT=709206ac8788475728cc9c992c25fb5f1501bc29

# For Pavics-landing notebooks to re-create Jupyter env layout:
# /notebook_dir for Pavics-landing notebooks to re-create Jupyter env layout:
# /notebook_dir/writable-workspace, /notebook_dir/pavics-homepage.
#
# Path to the /notebook_dir/pavics-homepage/tutorial_data/*.geojson files are
# hardcoded so users can copy the nb to writable-workspace/ dir and still be able
# to run them seemlessly from the Jupyter env (without having to also copy
# those *.geojson files with the notebooks).
RUN mkdir /notebook_dir && chown jenkins /notebook_dir
RUN wget https://raw.githubusercontent.com/jupyter/docker-stacks/$DOCKER_STACKS_COMMIT/base-notebook/start.sh --output-document /usr/local/bin/start.sh && \
wget https://raw.githubusercontent.com/jupyter/docker-stacks/$DOCKER_STACKS_COMMIT/base-notebook/start-singleuser.sh --output-document /usr/local/bin/start-singleuser.sh && \
wget https://raw.githubusercontent.com/jupyter/docker-stacks/$DOCKER_STACKS_COMMIT/base-notebook/start-notebook.sh --output-document /usr/local/bin/start-notebook.sh && \
wget https://raw.githubusercontent.com/jupyter/docker-stacks/$DOCKER_STACKS_COMMIT/base-notebook/fix-permissions --output-document /usr/local/bin/fix-permissions && \
mkdir /etc/jupyter && \
wget https://raw.githubusercontent.com/jupyter/docker-stacks/$DOCKER_STACKS_COMMIT/base-notebook/jupyter_notebook_config.py --output-document /etc/jupyter/jupyter_notebook_config.py && \
chmod a+rx /usr/local/bin/start.sh /usr/local/bin/start-singleuser.sh /usr/local/bin/start-notebook.sh /usr/local/bin/fix-permissions && \
chmod a+r /etc/jupyter/jupyter_notebook_config.py && \
mkdir /notebook_dir && chown jenkins /notebook_dir

# problem running start-notebook.sh when being root
# the jupyter/base-notebook image also do not default to root user so we do the same here
Expand Down
6 changes: 3 additions & 3 deletions docker/Dockerfile.testing
View file
@@ -1,15 +1,15 @@
# For testing quickly without having to do a full rebuild.

FROM pavics/workflow-tests:211123
FROM pavics/workflow-tests:211221

USER root

# Use 'update' for existing and 'install' for new package.
# Keep same channel ordering to not revert anything.
RUN umask 0000 \
&& conda install -c conda-forge -c cdat -c bokeh -c plotly -c defaults -n birdy "rasterio<=1.2.6" "ravenpy>=0.7.5"
&& mamba install -c conda-forge -c cdat -c bokeh -c plotly -c defaults -n birdy "shapely<=1.7.1" "bokeh<=2.3.3"
# && pip uninstall -y ravenpy \
# && conda install -c conda-forge -c cdat -c bokeh -c plotly -c defaults -n birdy ravenpy aiohttp
# && mamba install -c conda-forge -c cdat -c bokeh -c plotly -c defaults -n birdy ravenpy aiohttp

# RUN apt-get update && \
# DEBIAN_FRONTEND=noninteractive apt-get install -y unzip && \
Expand Down
92 changes: 66 additions & 26 deletions docker/environment.yml
View file
Expand Up @@ -6,27 +6,39 @@ channels:
- bokeh
- plotly # for jupyter-dash
- defaults

dependencies:

# Do not put xclim and ravenpy direct dependencies here to let xclim and ravenpy
# manage their own dependencies pinning.
#
# xclim direct dependencies: https://github.com/conda-forge/xclim-feedstock/blob/master/recipe/meta.yaml
# ravenpy direct dependencies: https://github.com/conda-forge/ravenpy-feedstock/blob/master/recipe/meta.yaml

# Pin latest xclim and ravenpy to avoid downgrading during the second install
# phase. Mamba is quicker to solve dependencies than conda but it is less
# precise so accidental downgrade happends.
- xclim >= 0.32.1
- ravenpy >= 0.7.8

- matplotlib
- xarray
- numpy
# - xarray # from xclim and ravenpy
# - numpy # from xclim and ravenpy
- birdy
- owslib>=0.23.0
- netcdf4
# - owslib>=0.23.0 # from ravenpy
# - netcdf4 # from ravenpy
# https://github.com/ecmwf/cfgrib
# Python interface to map GRIB files to the Unidata's Common Data Model v4
# following the CF Conventions.
- cfgrib
- pydap
- cartopy
- cartopy >= 0.20.1
- descartes
# Pin rasterio for ravenpy, remove on next build.
# See https://github.com/CSHS-CWRA/RavenPy/commit/eae66e9afc30e2381e9119644a0695d1d248c739
- rasterio <= 1.2.6
- gdal # for osgeo
- geopandas
- pandas
- rioxarray
# - rasterio # from ravenpy
# - gdal # for osgeo, from ravenpy
# - geopandas # from ravenpy
# - pandas # from xclim and ravenpy
# - rioxarray # from ravenpy
- scikit-image
- ipyleaflet
- threddsclient
Expand All @@ -48,12 +60,12 @@ dependencies:
# pinning hvplot did not solve the problem with violin plot.
- hvplot
- nc-time-axis
- cftime
- statsmodels # for ravenpy
# - cftime # from xclim and ravenpy
# - statsmodels # for ravenpy
# for error 'ImportError: HTTPFileSystem requires "requests" and "aiohttp" to
# be installed' with call 'fsspec.filesystem('https')'
- aiohttp
- pydantic
# - pydantic # from ravenpy
# Intake is a lightweight set of tools for loading and sharing data in data science projects
- intake
# https://intake.readthedocs.io/en/latest/plugin-directory.html
Expand All @@ -74,27 +86,46 @@ dependencies:
- zarr
# https://github.com/dask/s3fs/
- s3fs
- xclim
# Pinning shapely for ravenpy. Remove on next rebuild.
# https://github.com/CSHS-CWRA/RavenPy/blob/f63e1e5b967c0d7c17e679c8f9d6d309a94096e6/environment.yml#L35
- shapely <=1.7.1
- ravenpy
# - shapely # from ravenpy
# PIN shapely due to notebook failure
# PAVICStutorial_ClimateDataAnalysis-5Visualization.ipynb:
# /opt/conda/envs/birdy/lib/python3.7/site-packages/shapely/geometry/base.py in array_interface_base(self)
# 324 "removed in Shapely 2.0.",
# 325 ShapelyDeprecationWarning, stacklevel=2)
# --> 326 return self._array_interface_base()
# 327
# 328 @property
#
# TypeError: 'dict' object is not callable
#
# climex.ipynb:
# /opt/conda/envs/birdy/lib/python3.7/site-packages/cartopy/crs.py:825: ShapelyDeprecationWarning: __len__ for multi-part geometries is deprecated and will be removed in Shapely 2.0. Check the length of the `geoms` property instead to get the number of parts of a multi-part geometry.
# if len(multi_line_string) > 1:
# /opt/conda/envs/birdy/lib/python3.7/site-packages/cartopy/crs.py:877: ShapelyDeprecationWarning: Iteration over multi-part geometries is deprecated and will be removed in Shapely 2.0. Use the `geoms` property to access the constituent parts of a multi-part geometry.
# for line in multi_line_string:
# /opt/conda/envs/birdy/lib/python3.7/site-packages/cartopy/crs.py:944: ShapelyDeprecationWarning: __len__ for multi-part geometries is deprecated and will be removed in Shapely 2.0. Check the length of the `geoms` property instead to get the number of parts of a multi-part geometry.
# if len(p_mline) > 0:
# /opt/conda/envs/birdy/lib/python3.7/site-packages/cartopy/io/__init__.py:241: DownloadWarning: Downloading: https://naturalearth.s3.amazonaws.com/10m_physical/ne_10m_coastline.zip
# warnings.warn(f'Downloading: {url}', DownloadWarning)
- shapely <= 1.7.1
# https://github.com/roocs/clisops
- clisops
- clisops >= 0.8.0
# Universal Regridder for Geospatial Data
# https://github.com/pangeo-data/xESMF
# Pin xesmf because latest xesmf-0.6.2 is not compatible with latest clisops-0.7.0.
# xesmf-0.6.1 is buggy.
# Unpin when new compatible clisops is released.
- xesmf <= 0.6.0
# xesmf-0.6.2 requires clisops>=0.8.0
- xesmf >= 0.6.2
# https://anaconda.org/anaconda/memory_profiler
# Monitor memory consumption of a process as well as line-by-line analysis
# of memory consumption for Python programs.
- memory_profiler
# for esgf notebooks
- esgf-compute-api
- cdms2
- vcs
# Disable vcs because it was forcing python downgrade to below 3.9.
# See https://github.com/CDAT/vcs/issues/457
# package vcs-8.1-py_0 requires vtk-cdat >8.1, but none of the providers can be installed
# package vtk-cdat-8.2.0.8.2.1-py38hbc81915_0 requires python >=3.8,<3.9.0a0 *_cpython
#- vcs
- mesalib
# tests
- pytest
Expand All @@ -105,6 +136,10 @@ dependencies:
- notebook
- jupyterlab
- jupyterhub
# https://ipywidgets.readthedocs.io/en/latest/user_install.html
- ipywidgets
# https://github.com/jupyter-widgets/ipyleaflet
- ipyleaflet
# https://github.com/mamba-org/gator (was jupyter_conda)
- mamba_gator
# to diff .ipynb files
Expand All @@ -125,6 +160,11 @@ dependencies:
# xeus-python: back-end kernel implementing the Jupyter Debug Protocol
- xeus-python
- jupyter-dash
# Force newer nodejs for 'jupyter lab build' issue
# https://github.com/jupyterlab/jupyterlab/issues/11726#issuecomment-998901247
# TODO: remove nodejs once all extensions move to prebuilt extensions, see comment
# https://github.com/jupyterlab/jupyterlab/issues/11726#issuecomment-998917305
- nodejs >= 16.0
# utilities
- curl
- wget
Expand Down
2 changes: 1 addition & 1 deletion launchcontainer
View file
@@ -1,7 +1,7 @@
#!/bin/sh -x

if [ -z "$DOCKER_IMAGE" ]; then
DOCKER_IMAGE="pavics/workflow-tests:211123-update211216"
DOCKER_IMAGE="pavics/workflow-tests:220121"
fi

if [ -z "$CONTAINER_NAME" ]; then
Expand Down
2 changes: 1 addition & 1 deletion launchnotebook
View file
Expand Up @@ -7,7 +7,7 @@ if [ -z "$PORT" ]; then
fi

if [ -z "$DOCKER_IMAGE" ]; then
DOCKER_IMAGE="pavics/workflow-tests:211123-update211216"
DOCKER_IMAGE="pavics/workflow-tests:220121"
fi

if [ -z "$CONTAINER_NAME" ]; then
Expand Down

0 comments on commit 607ec58

Please sign in to comment.