All Cookbook builds currently broken

[brian-rose]

As of four days ago, it looks like all Cookbook including this Template are failing their nightly builds.

See build log here where this error is raised:

pkg_resources.ContextualVersionConflict: (jsonschema 3.2.0 (/usr/share/miniconda3/envs/cookbook-dev/lib/python3.10/site-packages), Requirement.parse('jsonschema[format-nongpl]>=4.3.0'), {'jupyter-events'})

[brian-rose]

Trying to run jupyter-book build locally on my laptop with a fresh environment yielded the same error.

[brian-rose]

@ProjectPythia/infrastructure hopefully we can get this fixed this week given the AMS meeting coming up.

[brian-rose]

Possibly related to ProjectPythia/intake-cookbook#26?

[brian-rose]

Interestingly, the Foundations book (which uses a very similar stack of dependencies) is still passing its nightly builds: https://github.com/ProjectPythia/pythia-foundations/actions/runs/3826152298/jobs/6509714548

[jnmorley]

I tried cloning the radar-cookbook, which seemed to be having this issue. After creating the Conda environment, I was not able to start Jupyter Lab. It would throw a ModuleNotFoundError: No module named 'jsonschema.protocols'. Comparing the Conda environment with a working environment I created a few days ago, I noticed that jupyter_server had updated from version 1.23.4 to version 2.0.6. Downgrading jupyter_server back to 1.23.4 seemed to fix the issue. I am not sure exactly what the problem is, but downgrading jupyter_server might work as a temporary work around for this issue. From what I can tell, jupyter_server version 2.0.6 tries to use a module in jsonschema that was added around jsonschema=4.3.0, but Jupyter Book requires a version of jsonschema <4.0.0

[brian-rose]

In #101 I added

- jupyter_server<2

to the environment.yml file and that resolved the build problem.

This does seem like a bandaid solution since there's nothing Pythia-specific that triggered this error. I think it's worth tracing the problem upstream and seeing where we can report it, or if it's already being tracked somewhere.

[jukent]

Should I go through and add this dependency to the other cookbooks?

[brian-rose]

Should I go through and add this dependency to the other cookbooks?

Yes, please @jukent that would be great.

If you have time, it would also be great if you could do some digging to see if this problem has been reported anywhere, e.g. jupyterlab or jupyterbook repos.

[mgrover1]

I suspect these issues are related to one that was opened ~13 hours ago jupyter-book/jupyter-book#1902

[jukent]

They all look to be passing to me now? I didn't make any changes.

[brian-rose]

They all look to be passing to me now? I didn't make any changes.

The cookbooks that are building via binder are not currently affected by this issue because they are using an earlier image cached on the binder. I assume they would probably fail with the same issue if we tried to build a new image with a new environment. Unfortunately we don't have a good way to test this right now.

[jukent]

I'm not sure I fully understand - but it sounds like all the Cookbooks displayed in the gallery are unaffected (except maybe the intake one).

[brian-rose]

I'm not sure I fully understand - but it sounds like all the Cookbooks displayed in the gallery are unaffected (except maybe the intake one).

The Radar cookbook was affected, but it was already fixed this morning in ProjectPythia/radar-cookbook#84

[brian-rose]

There are several gaps in our nightly testing strategy. One of them is this inconsistency between what gets tested when we execute the notebooks on GitHub Actions vs on the Binder. On Github Actions, we make a fresh new conda environment every calendar day, or if there are any changes to the environment file. On Binder, we use the cached image associated with the main branch of the repo. This means that the tests run much quicker since we don't have to solve a new environment and build a new docker image, but in some cases (like right now) we are then running the notebooks in a different, out-of-date environment relative to what conda would create for us today -- usually because some new package versions became available upstream. It's a bit of a double-edged sword -- the cached Binder image means that we can run and test the content of our notebooks, and keep doing that even when something breaks upstream -- but we're not testing the reproducibility of the environment.

Something to think about in the bigger picture of what our health checking suite should ideally cover.

[jukent]

Thanks for explaining. Topic for next IWG?

[clyne]

Thanks @brian-rose, @jukent, @jnmorley, @mgrover1 for unraveling this.

[brian-rose]

This was left open in order to improve our testing strategy to do a better job of catching this type of failure. We'll continue that discussion elsewhere. Closing this now.