Split ServiceManager into multiple plugins

[jtpio]

Problem

Currently a JupyterLab application needs to instantiate a single ServiceManager before startup. This service manager is then accessible via the JupyterFronEnd.IShell interface. Core plugins and extensions can then access it via app.serviceManager.

A ServiceManager allows accessing server components and features such as contents, kernels, sessions, nbconvert:

jupyterlab/packages/application/src/frontend.ts

Lines 108 to 111 in 897418f

	/**
	* The service manager used by the application.
	*/
	readonly serviceManager: ServiceManager.IManager;

However it is currently not possible to easily and partially replace some of its underlying services such as the kernel or session managers. Custom applications need to instantiate their own ServiceManager if they want to tweak the behavior of some of the internals.

Proposed Solution

Probably the main reason for having the ServiceManager be a "special" component of a JupyterLab application is likely based on the assumption that a JupyterFronEnd should be served via a fully-featured Jupyter Server (or even a JupyterLab Server).

However over the past years we have seen more use cases that are starting to challenge this assumption. For example distributions with in-browser kernels like JupyterLite, or plugin-based Jupyter Server alternatives like Jupyverse, which use or enable only a subset of what a traditional Jupyter Server offers.

In these situations a plugin-based ServiceManager would allow for more easily disable or replace some services via normal extensions. This would mean providing each service of the ServiceManager as a token, for example IKernelManager or ISessionManager.

One challenge could be to keep backward compatibility with all existing extensions that access the contents, session, kernel managers via app.serviceManager. This could likely be achieved and would avoid existing extensions to have to require new tokens.

Additional context

For reference there was a previous attempt at doing something similar: #13558. This PR was focused on replacing the whole ServiceManager via a plugin, but was still treated as a special case.

This would allow for some use cases.

Allow extending the kernel / service / session manager using an extension

Which could help fix: #12504

Custom lab distributions

Custom lab distributions like JupyterLite that currently need to redefine the whole ServiceManager to allow for custom kernel and contents managers. Splitting the ServiceManager into multiple plugins would allow for such applications to just be regular JupyterLab extensions instead.

Remote content and kernels

Users and administrators could more easily combine extensions, aggregate content from remote locations and use custom kernel managers as well.

[jtpio]

Adding to 5.0.0 for consideration.

[echarles]

How different this would be from the discussions that occured in. #13558?

[jtpio]

How different this would be from the discussions that occured in. #13558?

See above:

For reference there was a previous attempt at doing something similar: #13558. This PR was focused on replacing the whole ServiceManager via an plugin, but was still treated as a special case.

In these situations a plugin-based ServiceManager would allow for more easily disable or replace some services via normal extensions. This would mean providing each service of the ServiceManager as a token, for example IKernelManager or ISessionManager.

So instead of treating the ServiceManager as a special case, all of its underlying services are provided via plugins. Also suggested at the end of the discussion in #13558 (comment).

[jtpio]

There might of course be some challenges, but this issue suggests taking a look at it to see how feasible this would be.