Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Jupyter env: new full build with latest of almost everything #453

Merged
merged 4 commits into from May 11, 2024
Merged

Jupyter env: new full build with latest of almost everything #453

merged 4 commits into from May 11, 2024

Conversation

tlvu
Copy link
Collaborator

@tlvu tlvu commented May 8, 2024

See Ouranosinc/PAVICS-e2e-workflow-tests#121 for more info.

birdhouse_daccs_configs_branch: master
birdhouse_skip_ci: false

@github-actions github-actions bot added component/jupyterhub Related to JupyterHub as development frontend with notebooks documentation Improvements or additions to documentation labels May 8, 2024
@tlvu tlvu mentioned this pull request May 8, 2024
Merged
tlvu added a commit to Ouranosinc/PAVICS-e2e-workflow-tests that referenced this pull request May 9, 2024
@tlvu
Jupyter docker: new full build with latest of almost everything excep…
c7af8b8 
…t xclim and ravenpy to smooth transition (#121)

# Overview

This new full build has latest of almost everything except `xclim` and
`ravenpy` as intermediate step to smooth transition to `pandas` 2.2 freq
strings changes.

## Changes

- New: save conda env export, DockerHub build logs and Jenkins test
result in the repo to track changes much more easily between releases

- Jenkins: add `SAVE_RESULTING_NOTEBOOK_TIMEOUT` for slow notebooks or
slow machine

- Jupyter env changes:
- add `conda-pack` so we can export the conda env outside of the docker
image if need to run locally without docker
  - upgrade from Python 3.9 to 3.11
  - Relevant changes (alphabetical order):
```diff
-  - birdy=0.8.4=pyh1a96a4e_0
+      - birdhouse-birdy==0.8.7

# major upgrade from v2 to v3
-  - bokeh=2.4.3=pyhd8ed1ab_3
+  - bokeh=3.4.1=pyhd8ed1ab_0

-  - cartopy=0.21.1=py39h6e7ad6e_0
+  - cartopy=0.23.0=py311h320fe9a_0

-  - cf_xarray=0.8.0=pyhd8ed1ab_0
+  - cf_xarray=0.9.0=pyhd8ed1ab_0

-  - cfgrib=0.9.10.4=pyhd8ed1ab_0
+  - cfgrib=0.9.11.0=pyhd8ed1ab_0

-  - cftime=1.6.2=py39h2ae25f5_1
+  - cftime=1.6.3=py311h1f0f07a_0

-  - climpred=2.3.0=pyhd8ed1ab_0
+  - climpred=2.4.0=pyhd8ed1ab_0

-  - clisops=0.9.6=pyh1a96a4e_0
+  - clisops=0.13.0=pyhca7485f_0

-  - dask=2023.5.1=pyhd8ed1ab_0
+  - dask=2024.5.0=pyhd8ed1ab_0

-  - geopandas=0.13.0=pyhd8ed1ab_0
+  - geopandas=0.14.4=pyhd8ed1ab_0

-  - hvplot=0.8.3=pyhd8ed1ab_0
+  - hvplot=0.9.2=pyhd8ed1ab_0

-  - numpy=1.23.5=py39h3d75532_0
+  - numpy=1.24.4=py311h64a7726_0

-  - numba=0.57.0=py39hb75a051_1
+  - numba=0.59.1=py311h96b013e_0

# major upgrade from v1 to v2
-  - pandas=1.3.5=py39hde0f152_0
+  - pandas=2.1.4=py311h320fe9a_0

# major upgrade to v1
-  - panel=0.14.4=pyhd8ed1ab_0
+  - panel=1.4.2=pyhd8ed1ab_0

# major upgrade from v1 to v2
-  - pydantic=1.10.8=py39hd1e30aa_0
+  - pydantic=2.7.1=pyhd8ed1ab_0

# Python 3.9 to 3.11
-  - python=3.9.16=h2782a2a_0_cpython
+  - python=3.11.6=hab00c5b_0_cpython

-  - raven-hydro=0.2.1=py39h8e2dbb5_1
+  - raven-hydro=0.2.4=py311h64a4d7b_0

-  - ravenpy=0.12.1=py39hf3d152e_0
+      - ravenpy==0.13.1

-  - rioxarray=0.14.1=pyhd8ed1ab_0
+  - rioxarray=0.15.5=pyhd8ed1ab_0

-  - roocs-utils=0.6.4=pyh1a96a4e_0
+  - roocs-utils=0.6.8=pyhd8ed1ab_0

-  - scipy=1.9.1=py39h8ba3f38_0
+  - scipy=1.13.0=py311h517d4fd_1

-  - xarray=2023.1.0=pyhd8ed1ab_0
+  - xarray=2023.8.0=pyhd8ed1ab_0

-  - xclim=0.43.0=py39hf3d152e_1
+  - xclim=0.47.0=py311h38be061_0

-  - xesmf=0.7.1=pyhd8ed1ab_0
+  - xesmf=0.8.5=pyhd8ed1ab_0

-  - xskillscore=0.0.24=pyhd8ed1ab_0
+  - xskillscore=0.0.26=pyhd8ed1ab_0

+  - xscen=0.8.2=pyhd8ed1ab_0

+      - figanos==0.3.0

-      - xncml==0.2
+      - xncml==0.4.0

```


## Test

- Deployed as "beta" image in production for bokeh visualization
performance regression testing.
- Manual test notebook
https://github.com/Ouranosinc/PAVICS-landing/blob/master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-5Visualization.ipynb
for bokeh visualization performance and it looks fine.
- Jenkins build:
- Default notebooks, all passed:
https://github.com/Ouranosinc/PAVICS-e2e-workflow-tests/blob/54792e6510adfcd1bb21e1bd31fdfa36c5c634e0/docker/saved_buildout/jenkins-buildlogs-default.txt
- Raven notebooks, only known `HydroShare_integration.ipynb` failing:
https://github.com/Ouranosinc/PAVICS-e2e-workflow-tests/blob/931cfc924a147d07b59e88badff9f170e852a03b/docker/saved_buildout/jenkins-buildlogs-raven.txt


## Related Issue / Discussion

- Matching notebook fixes:
  - Pavics-sdi: PR Ouranosinc/pavics-sdi#321
  - Finch: PR url: None
- PAVICS-landing: PR
Ouranosinc/PAVICS-landing#78
  - RavenPy: PR CSHS-CWRA/RavenPy#356
  - Resolves Ouranosinc/PAVICS-landing#65
  - Resolves Ouranosinc/PAVICS-landing#66

- Deployment to PAVICS:
bird-house/birdhouse-deploy#453

- Jenkins-config changes for new notebooks: PR url: None

- Other issues found while working on this one
  - computationalmodelling/nbval#204
  - jupyterlab-contrib/jupyter-archive#132
  - CSHS-CWRA/RavenPy#357
  - CSHS-CWRA/RavenPy#361
  - CSHS-CWRA/RavenPy#362

- Previous release: PR
#134


## Additional Information

Full diff conda env export:

81deb99...931cfc9#diff-e8f2a6a53085ae29bb7cedc701c1d345a330651ae971555e85a5c005e94f4cd9


Full new conda env export:

https://github.com/Ouranosinc/PAVICS-e2e-workflow-tests/blob/931cfc924a147d07b59e88badff9f170e852a03b/docker/saved_buildout/conda-env-export.yml


DockerHub build log

https://github.com/Ouranosinc/PAVICS-e2e-workflow-tests/blob/931cfc924a147d07b59e88badff9f170e852a03b/docker/saved_buildout/docker-buildlogs.txt
fmigneault
fmigneault reviewed May 9, 2024
View reviewed changes
CHANGES.md Outdated Show resolved Hide resolved
@tlvu tlvu requested a review from fmigneault May 9, 2024 21:07
mishaschwartz
mishaschwartz requested changes May 10, 2024
View reviewed changes
Copy link
Collaborator

@mishaschwartz mishaschwartz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. Can we please also update the reference in the comment here:

birdhouse-deploy/birdhouse/env.local.example

Line 266 in a095db4

#export DOCKER_NOTEBOOK_IMAGES="pavics/workflow-tests:210216 \

@tlvu
Copy link
Collaborator Author

tlvu commented May 10, 2024

Looks good. Can we please also update the reference in the comment here:

birdhouse-deploy/birdhouse/env.local.example

Line 266 in a095db4

#export DOCKER_NOTEBOOK_IMAGES="pavics/workflow-tests:210216 \

That one is just an example. Once user copy env.local.example to env.local, they probably will never look back at env.local.example so I don't think there is value in keeping env.local.example up-to-date.

As time goes by, the example value in their env.local will become outdated anyways and there is no way I can control their copy of env.local.

@tlvu tlvu requested a review from mishaschwartz May 10, 2024 15:10
@mishaschwartz
Copy link
Collaborator

I don't think there is value in keeping env.local.example up-to-date

New deployments will definitely use env.local.example as a reference to define which settings they will enable. If we're going to let it get out of date then it's not a good example at all. I think it is critical for new deployments that the env.local.example be kept up to date.

@tlvu
env.local.example: also update pavics/workflow-tests version
95591d8 
Added a comment to see components/jupyterhub/default.env because once
env.local.example is copied to env.local, that sample version of
pavics/workflow-tests won't be up-to-date anymore.  We can not control
user's env.local copy.
@tlvu
Copy link
Collaborator Author

tlvu commented May 10, 2024

I think it is critical for new deployments that the env.local.example be kept up to date.

Sure, I am happy to comply, this is a simple change.

I also added "See components/jupyterhub/default.env for the most updated version of pavics/workflow-tests." so it is clear in their own copy of env.local that their value will eventually be not up-to-date anymore.

@github-actions github-actions bot added the ci/operations Continuous Integration components label May 11, 2024
@tlvu tlvu merged commit 8038856 into master May 11, 2024
3 of 4 checks passed
@tlvu tlvu deleted the update-jupyter-env branch May 11, 2024 03:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mishaschwartz mishaschwartz mishaschwartz approved these changes

@fmigneault fmigneault fmigneault approved these changes

Assignees
No one assigned
Labels
ci/operations Continuous Integration components component/jupyterhub Related to JupyterHub as development frontend with notebooks documentation Improvements or additions to documentation
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@tlvu @mishaschwartz @fmigneault