Support server-side execution

[davidbrochart]

References

Closes #2833.
Closes #12867.

Needs:

• Set file ID on document state jupyter-collaboration#217
• Add optional file ID to document state jupyter-server/jupyter_ydoc#198
• Change notebook code cell stream output schema jupyter-server/jupyter_ydoc#201
• jupyter-collaboration >=2.0
• jupyverse >=0.4.0

Currently, the only Jupyter server that supports this execution mode is Jupyverse.

Code changes

A new server-side-execution flag allows to execute a notebook server-side (false by default). In this execution mode, the frontend doesn't use the kernel protocol over WebSocket, but interacts with the notebook only through its shared model. Thus, this mode needs jupyter-collaboration.

User-facing changes

The user can close their JupyterLab browser window, reopen it later and recover the notebook state, including (live) cell outputs.

Backwards-incompatible changes

None.

[jupyterlab-probot]

Thanks for making a pull request to jupyterlab!
To try out this branch on binder, follow this link:

[matthewwiese]

This is really great David! So cool to follow your progress on this. 🙂

[davidbrochart]

Here is a demo:

Peek.2023-11-28.16-25.mp4

[krassowski]

server-side-execution

While I see why you named it "server-side-execution", I wonder if naming it "server-side-composition" could potentially avoid confusion with the execution which happens in kernels? It feels like the fact that execution is triggered from server (well, really proxied) is an implementation detail and what users care about is where the execution results are composed into the document.

[davidbrochart]

I agree that "server-side-execution" is not great, because kernels always execute code in the server. The main difference here is that:

• execution is triggered by a higher-level protocol (an HTTP POST instead of the frontend talking to the kernel through the kernel protocol over WebSocket), and the backend is responsible for talking to the kernel through the kernel protocol over ZMQ.
• cell outputs are not populated by the frontend, but by the backend, and the frontend sees the output changes reflected in their shared model.

But I'm not sure about "server-side-composition" either. Maybe "server-side-outputs"?

[jtpio]

but interacts with the notebook only through its shared model. Thus, this mode needs jupyter-collaboration.

Curious if this could be implemented in jupyter-collaboration directly then? Just trying to see if this implicit dependency could create issues in the future since jupyterlab and jupyter-collaboration evolve at a different pace.

[davidbrochart]

The execution command is done through an HTTP POST instead of using the kernel protocol, so there are changes to be done in JupyterLab:

jupyterlab/packages/notebook/src/actions.tsx

Lines 2480 to 2500 in 21cc49a

	if (PageConfig.getOption('serverSideExecution') === 'true') {
	const kernelId = sessionContext.session?.kernel?.id;
	const settings = ServerConnection.makeSettings();
	const apiURL = URLExt.join(
	settings.baseUrl,
	`api/kernels/${kernelId}/execute`
	);
	const cellId = cell.model.sharedModel.getId();
	const documentId = `json:notebook:${notebook.model?.sharedModel.getState(
	'file_id'
	)}`;
	const body = JSON.stringify({
	cell_id: cellId,
	document_id: documentId
	});
	const init = {
	method: 'POST',
	body
	};
	ServerConnection.makeRequest(apiURL, init, settings);
	ran = true;

But I'm not sure it's the right place.

[davidbrochart]

Here is a way to try it out:

mkdir jupyterlab
cd jupyterlab
git clone https://github.com/davidbrochart/jupyterlab.git
git clone https://github.com/davidbrochart/jupyverse.git
pixi init
pixi add nodejs pip "python<3.12"
pixi shell

cd jupyterlab
git checkout server-side-execution
pip install -e .
cd ..

cd jupyverse
git checkout ywidget-subdoc
pip install -e jupyverse_api
pip install -e plugins/noauth
pip install -e plugins/contents
pip install -e plugins/frontend
pip install -e plugins/lab
pip install -e plugins/jupyterlab
pip install -e plugins/kernels
pip install -e plugins/yjs
pip install -e plugins/nbconvert
pip install -e plugins/terminals
pip install -e .
cd ..

pip install ypywidgets==0.6.3
pip install jupyter-collaboration

mkdir tmp
cd tmp
jupyverse --set jupyterlab.dev_mode=true --set jupyterlab.server_side_execution=true --set kernels.require_yjs=true

Open a browser window at http://127.0.0.1:8000, it should open JupyterLab.
Create a notebook with the following cell and run it:

from time import sleep

for i in range(1000000):
    print(i)
    sleep(2)

Close your browser window and open a new one at http://127.0.0.1:8000. Open the notebook, you should see the live outputs.

[davidbrochart]

Actually, this PR only needs jupyter-server/jupyverse#364. Since having jupyter-server support this execution mode will require a significant amount of work and time, I think I could make a release of Jupyverse and we could merge this PR, what do you think?

[echarles]

will require a significant amount of work and time,

I will have a look next week at that amount of work and time to get that feature in jupyter-server and will report back. This will also be useful to asses that this PR would also work with jupyter-server. In the meantime, I propose to wait before merging.

[jtpio]

Also still curious if this could be implemented in jupyter-collaboration directly (#15448 (comment))? If it's not, maybe we can provide an extension point to do it as an alternative?

[davidbrochart]

I will have a look next week at that amount of work and time to get that feature in jupyter-server

There are changes not only in jupyter-server, but also in jupyter-client. I'm estimating at least one month of work.

This will also be useful to asses that this PR would also work with jupyter-server.

This PR is just replacing the kernel execute_request with an HTTP POST /api/kernels/{kernelId}/execute, under a new server-side-execution flag. It's really transparent for users who don't care about the new feature, but users interested in notebook state recovery will love it. I think it would be nice to release it early, so that users can try it out. We can also get feedback so that all the details are ironed out when we want to implement it in jupyter-server.

[davidbrochart]

Also still curious if this could be implemented in jupyter-collaboration directly (#15448 (comment))?

No it cannot, as I said JupyterLab must do an HTTP POST instead of using the kernel protocol.

If it's not, maybe we can provide an extension point to do it as an alternative?

Where would this extension point be?

[echarles]

There are changes not only in jupyter-server, but also in jupyter-client. I'm estimating at least one month of work.

Let me look at that, I will report asap.

[jtpio]

Where would this extension point be?

Currently there is no such extension point, as the notebook execution logic seems to be living in a Private namespace which makes it difficult to customize indeed. But ideally we could imagine having a plugin that would allow for using a different execution mode (WebSocket or HTTP POST), that would be made available to the notebook actions.

[davidbrochart]

Yes I was looking for something like that, but eventually took a shortcut. What I like in the current solution is that it's simple and it also doesn't drop the connection to the kernel through the kernel protocol over WebSocket, which means that the kernel busy indicator still works, and widgets should also still work (although they won't be recovered from the notebook state).
I see this PR as an incremental step towards completely getting rid of the kernel protocol over WebSocket in the frontend, which will require more work.

[davidbrochart]

Yes, in order to execute a cell of a notebook, the notebook document ID and the notebook cell ID are needed.
Support for client-side execution is still there, since the new server-side execution mode is under a flag.

[davidbrochart]

I will have a look next week at that amount of work and time to get that feature in jupyter-server and will report back. This will also be useful to asses that this PR would also work with jupyter-server. In the meantime, I propose to wait before merging.

What were your conclusions @echarles ?

[jtpio]

But ideally we could imagine having a plugin that would allow for using a different execution mode (WebSocket or HTTP POST), that would be made available to the notebook actions.

FYI @fcollonval just opened a PR to allow for something like this: #15830. Thanks Fred!

[EduardDurech]

Does anything still need to be done to merge this? I can give some time as this is critical and still the top 👍 issue after many years

In the meantime until the next release, can we just build locally and run this to get the functionality?

[davidbrochart]

I'm sorry this is taking so long, I'd love it to be released in JupyterLab 4.2.
But yes building JupyterLab with this PR is all that is needed to try it out, since Jupyverse has been released with the feature.

[jtpio]

#15830 is likely the way forward with this, with jupyter-collaboration providing its own ICellExecutor.

[davidbrochart]

Sure, #15830 is the right solution long-term, but maybe we could live with the shortcut provided by this PR in the meantime?

[jtpio]

#15830 is almost ready and also planned for 4.2.0. So I'm not sure there is a need to merge this PR in the meantime.

Once #15830 is merged and available in a 4.2 pre-release, the logic in this PR can be implemented in jupyter-collaboration directly instead.

[krassowski]

@davidbrochart FYI #15830 is now merged.

[davidbrochart]

Thanks, I'll update this PR.

[krassowski]

@davidbrochart is the plan to add a new implementation of ICellExecutor as a new plugin in this PR or to add it in jupyter-collaboration for time being and possibly bring the non-RTC bits of jupyter-collaboration into the core in the next release, say 4.3 (we are already past feature freeze on 4.2)?

[davidbrochart]

You're right, I think a new implementation of ICellExecutor as a new plugin should be done in jupyter-collaboration, as this execution mode doesn't make sense in non-RTC mode.

[davidbrochart]

Closing in favor of jupyterlab/jupyter-collaboration#279.