Update to JupyterLab 4 and Notebook 7

[jtpio]

References

Fixes #826
Fixes #925
Fixes #262
Closes #457

Code changes

• Update @jupyterlab and @lumino packages
• Remove a few extensions in the demo environment that are not compatible with JupyterLab 4 yet
• Adjust message passing based on Protocol alignment jupyterlab/jupyterlab#11841
• Update to Yarn 3: use jlpm for development
• Update to @jupyter-notebook (Notebook 7) packages
• Package Notebook 7 pre-releases on conda: Package Notebook 7 pre-releases conda-forge/notebook-feedstock#109
• Reuse upstream Playwright config, fix TODO added in Update to Playwright 1.29 #923
• Fix Galata activation: https://jupyterlab.readthedocs.io/en/latest/extension/extension_migration.html#testing-with-galata
• Closes [wip] Use appUrl and file name for shareable link #457
• Check Document in-app keyboard shortcuts #509
• Add @jupyterlab/lsp-extension
• Add @jupyterlab/metadataform-extension
• Disable @jupyterlab-apputils-extension:announcements
• Mock ws://localhost:8000/api/events/subscribe or disable events for now?
• Mock lsp/status endpoint
• Notebook 7:
  • Follow same plugin templating as in https://github.com/jupyter/notebook
  • Build multiple apps for each endpoint?
    • edit
    • consoles
    • notebooks
    • tree
• Fix TODO and use a custom document opener plugin:

jupyterlite/packages/retro-application-extension/src/index.ts

Lines 46 to 77 in fa5f44f

	/**
	* A plugin to open document in a new browser tab.
	*
	* TODO: remove and use a custom doc manager?
	*/
	const docmanager: JupyterFrontEndPlugin<void> = {
	id: '@jupyterlite/retro-application-extension:docmanager',
	requires: [IDocumentManager],
	autoStart: true,
	activate: (app: JupyterFrontEnd, docManager: IDocumentManager) => {
	const baseUrl = PageConfig.getBaseUrl();
	// patch the `docManager.open` option to prevent the default behavior
	const docOpen = docManager.open;
	docManager.open = (
	path: string,
	widgetName = 'default',
	kernel?: Partial<Kernel.IModel>,
	options?: DocumentRegistry.IOpenOptions
	): IDocumentWidget | undefined => {
	const ref = options?.ref;
	if (ref === '_noref') {
	docOpen.call(docManager, path, widgetName, kernel, options);
	return;
	}
	const ext = PathExt.extname(path);
	const route = ext === '.ipynb' ? 'notebooks' : 'edit';
	window.open(`${baseUrl}retro/${route}?path=${path}`);
	return undefined;
	};
	},
	};

• Fix UI tests:
  • Looks like some failures are related to Bump lumino packages jupyterlab/jupyterlab#14945. Updating to the latest 4.0.5 packages fixes it
  • Update Firefox snapshots
  • Update Chromium snapshots
  • Fix some tests
  • Re-add retries
  • Enable Playwright trace. Fixes Pipe the UI tests console messages to a file on CI #925
• Fix redirect with /lab, /tree ...
• CSS fixes:
  • Checkpoint indicator
  • File browser search bar
  • Tree page drop shadow
• Fix super slow and memory hungry builds. For example: https://github.com/jupyterlite/jupyterlite/actions/runs/5888491476/job/15969799639?pr=1019
  • force utf-8 encoding on the windows check
• Fix CSS duplication issue:

JupyterLab extensions to update to JupyterLab 4

• jupyterlab_miami_nights: JupyterLab 4 and build / CI tool uplift timkpaine/jupyterlab_miami_nights#57
• jupyterlab-drawio: support for JupyterLab 4.0 QuantStack/jupyterlab-drawio#119
• jupyterlab-github: Upgrade extension to JupyterLab 4 jupyterlab/jupyterlab-github#145
• jupyterlab-kernelspy: Update to JupyterLab 4 jupyterlab-contrib/jupyterlab-kernelspy#90
• jupyterlab-markup
• jupyterlab-myst: 🪐 Support for JupyterLab 4.x (take 2!) executablebooks/jupyterlab-myst#155
• theme-darcula

User-facing changes

Maybe user facing changes are similar to the JupyterLab 4 and Notebook 7 changes:

• https://jupyterlab.readthedocs.io/en/latest/getting_started/changelog.html#v4-0
• https://jupyter-notebook.readthedocs.io/en/latest/notebook_7_features.html

Backwards-incompatible changes

Same as JupyterLab 4 for extension authors.

Follow-ups

• Follow the same /tree, /edit, /notebooks as the regular Notebook, instead of keeping the notebook app under /notebook -> done ✔️
• Add workflow to update the playwright snapshots, like in the notebook repo: https://github.com/jupyter/notebook/blob/main/.github/workflows/playwright-update.yml -> added, need to be tested
• Rethink how to build the JupyterLite apps: bottom-up or top-down + disabledExtensions? (see also Add a check to ensure upstream plugins and extensions are included in the build #1022) -> tree, consoles, notebooks... are their own apps with disabledExtensions
• Improve integrity check to make sure plugins are correctly included (add script / ensure-repo check)

[github-actions]

👈 Try it on ReadTheDocs

[jtpio]

Opening early mostly to find potential issues with the lab 4 and notebook 7 pre-releases before they enter beta and rc phases.

Still planning to cut a 0.1.0 out soon, based on JupyterLab 3.5.x and RetroLab packages (#383).

[jtpio]

Made a couple of edits based on the suggestions above.

Some remaining suggestions are not strictly related to the update to the lab 4 and notebook 7 packages and could be addressed as follow-ups.

I would like to proceed with the merge of this PR and release a first 0.2.0a0 to unblock @trungleduc, @martinRenou and @hbcarlos who are waiting for packages to be published so they can update downstreams like Voici and RTC.

[bollwyvl]

unblock

Welp, we did set it up so they can already get started with a manual download of the wheels:

https://jupyterlite--1019.org.readthedocs.build/en/1019/_static/jupyterlite_core-0.1.2-py3-none-any.whl
https://jupyterlite--1019.org.readthedocs.build/en/1019/_static/jupyterlite-0.1.2-py3-none-any.whl

But of course all the npm .tgz are not available. Guess we could hoist those to RTD, too.

Will take a look at some more of the code changes, but from test driving on RTD:

Hard blockers:

• running kernels sidebar don't update
  • meaning there's no UI way to kill one
  • but probably indicative of something deeper
  • after opening a bunch of notebooks, the service worker basically dies
    • Uncaught (in promise) DOMException: Failed to execute 'put' on 'Cache': Quota exceeded.
    • at that point, you can't open/save anything anymore, even after reloading the page
      • requires a new (private) browser
• all notebooks are rendering with extra newlines
  • this actually breaks notebooks like ipycanvas, where there are long multiline strings
• sometimes saving (maybe just opening?) notebooks creates duplicates in the file browser

Core stuff:

• 404 for at least one config

_static/api/config/jupyterlabapputilsextensionannouncements

• mathjax missing fonts for rendered outputs (js notebook)

_static/lab/js/output/chtml/fonts/tex-woff-v2/MathJax_Zero.woff

• still appears to draw, but might have extra FLoUC
• cell toolbar appears to be throwing more errors than before

celltoolbartracker.js:239 Uncaught TypeError: Cannot read properties of undefined (reading 'node'

• toc is throwing errors

toc.js:486 Uncaught (in promise) TypeError: Cannot read properties of undefined (reading 'type')

Extension stuff:

• jupyterlab-kernelspy can probably be bumped to a compatible version
  • current one is broken in a deep lumino vdom spot, gets blank sidebar
  • if not working, i guess pull for now...
• altair notebook throws a bunch of deprecation warnings
• the ipyvue widgets don't have matching kernel/extension packages and appear about 50% broken
  • but still cost a few mb of downloads on page load, plus the 11mb of donwload at kernel start
• leaflet can't find mathjax
  • this appears to fire pretty early, so breaks more than just mathjax

[jtpio]

But of course all the npm .tgz are not available. Guess we could hoist those to RTD, too.

This is not going to help if the downstreams also want to make pre-releases.

Hard blockers:

Not sure they are all hard blockers for a first alpha release...

all notebooks are rendering with extra newlines

Yeah also noticed that one, which might look similar to jupyter/notebook#7002.

[jtpio]

Some of the Mathjax / ipywidgets issues might be related to jupyter-widgets/ipywidgets#3829

[jtpio]

running kernels sidebar don't update

That one might be related to jupyterlab/jupyterlab#14962, or jupyterlab/jupyterlab#13792 / jupyterlab/jupyterlab#13851.

[bollwyvl]

Some quick findings on RTD:

/tree/:

• opening/creating any file breaks on RTD (no trailing slash)

/notebook/:

• broken busy favicon
  • https://jupyterlite--1019.org.readthedocs.build/static/favicons/favicon-busy-1.ico

[jtpio]

Opened #1141 to track follow-ups and remaining issues.

I'll look into these issues right after. And also planning to make a first 0.2.0a0 release today to unblock other streams of work.

Thanks all for the reviews!