Test Notebook 7 on CI

[jtpio]

As a JupyterLab-based application, Notebook 7 is currently being developed in https://github.com/jupyter/notebook.

Notebook 7 depends on some of the core @jupyterlab extensions and plugins (but not all), and reassembles them in a different way to produce a Jupyter Frontend similar to what the Classic Notebook has been offering for many years.

However since it's a separate application, changes in JupyterLab must first be released before they can be integrated in Notebook 7. Often this is fine and updating to the latest @jupyterlab works without too much effort.

However sometimes changes in JupyterLab may end up affecting Notebook 7 later on. Here are some examples that were the result of back and forth between lab and notebook:

• Make IStatusBar optional for the notification plugin #14593
• Store the real position of the item in reactive toolbar #16111
• Do not call notifyCommandChanged for commands registered by other plugins #16113

Because Jupyter Notebook is an official Jupyter Frontend, it could be interesting to have a way to test changes that land in JupyterLab and see how they may affect downstream applications, and more precisely Notebook 7.

Some ideas:

• add a CI workflow on the JupyterLab repo to check if a PR would work with Notebook 7
• tests could be UI tests (or the Notebook 7 UI tests directly), just to get an idea if something might be wrong and worth looking at in a PR
• use yalc or similar to link JupyterLab packages in Notebook before building Notebook
• trigger the check only on-demand? (like for the benchmarks)

The main drawback of this approach would be added CI complexity and possibly extra maintenance effort. Also sometimes the tests will be expected to fail when something visual changes, and a new plugin is added.

It might also be worth taking some inspiration from jupyak, which allows testing PRs from various Jupyter repos.

[krassowski]

Canary testing with a single or two snapshots sounds good for me. Running all UI tests may be annoying.