FileSystem calls over Atomics.wait instead of service worker

[martinRenou]

In combination with jupyterlite/jupyterlite#1383

This PR provides an implementation of the Emscripten FileSystem that communicates using Atomics.wait and SharedArrayBuffer instead of the blocking HTTP requests to a service worker.

I open a PR here in jupyterlite/xeus for convenience, but part of the code here should probably go upstream in jupyterlite. Done in jupyterlite/jupyterlite#1383

How to test it:

1. Build your jupyterlite site as usual.
2. Serve it using the proper headers, so that shared array buffers can be used: npx static-handler --cors --coop --coep --corp ./

Implementation details

Instead of directly using SharedArrayBuffers and Atomic.waits (which would require defining a proper communication protocol), I decided to go with using https://github.com/WebReflection/coincident which provides high level APIs for making blocking communication between worker <-> main thread.

Coincident comes as a replacement for comlink which we used for wrapping the worker kernel communication.

[martinRenou]

I'll turn this into draft and leave it open for discussions. But I'll open the proper PR upstream in JupyterLite to introduce the needed APIs for this to work nicely.

[jtpio]

Nice, thanks @martinRenou for looking into this!

[martinRenou]

This is all working nicely (except ipyleaflet of course, because CORS are disabled...)

[jtpio]

We should probably document how to enable this. It could be here in the jupyterlite-xeus docs, or maybe in the main JupyterLite documentation directly, as we will probably want to support it in the Pyodide kernel too?

[martinRenou]

We should definitely look at enabling this for jupyterlite-pyodide too :)

Agreed we should add it to the docs.

[jtpio]

Testing locally: one advantage of this approach is that it works in a Firefox private window 👍

[jtpio]

Tested locally with and without the COOP / COEP headers to check it falls back to using the Service Worker 👍

Thanks for working on this!

[jtpio]

@martinRenou should this be released in a 0.1.9, or 0.2.0?

[martinRenou]

This should be backward compatible if not setting --cors --coop --coep --corp (actually the CI passed because it's backward compatible in that case, we're still using jupyterlite 0.3.x there).

If people were already setting --cors --coop --coep --corp, file access will eventually fail and crash the kernel due to https://github.com/jupyterlite/xeus/pull/87/files#diff-a0841713a169cff378e9935d6ae3ba34d57653c34fcea3294ac4771db5427b14R185

So I believe this is backward incompatible.

[martinRenou]

I believe we can't make it backward compatible though, so we should probably make an alpha for 0.2.0. Would you like to do it ?

[jtpio]

Yeah I was thinking a 0.2.0 would be on the safer side too.

Starting a 0.2.0-alpha.0 now.

[jtpio]

We should probably document how to enable this. It could be here in the jupyterlite-xeus docs, or maybe in the main JupyterLite documentation directly,

Started to document this in jupyterlite/jupyterlite#1405

[martinRenou]

I can have a look at implementing the shared array buffers approach for the pyodide kernel.

[jtpio]

I can have a look at implementing the shared array buffers approach for the pyodide kernel.

That would be great, thanks! Hopefully the approach will be similar.