Support WebWorker-based Language Servers

[bollwyvl]

References

• Multiple sources of LSP messages on frontend and backend jupyter-lsp/jupyterlab-lsp#184
• Get a language server working jupyterlite/jupyterlite-lsp#1

Problem

The current DocumentConnectionManager wraps WebSocket sessions from the upstream jupyter-lsp jupyter_server extension, which in turn manages the discovery and running of, and JSON-RPC communication with, heavyweight processes on the server.

Many of the widely-used Language Servers are written in JS/TypeScript, but run in nodejs, historically a difficult-to-manage runtime dependency for Jupyter interactive computing users.

Proposed Solution

However, many of these nodejs-based language Servers can be made to run in the browser. Allowing such servers to be independently packaged and distributed as JupyterLab extensions would provide additional editing support for non-kernel (or not-yet-kernel) content, without requiring a nodejs installation.

Of particular interest to JupyterLab itself is a JSON Schema-aware Language Server, which would allow for, at a minimum, improving the user experience of the JupyterLab JSON Settings Editor and the myriad JSON-adjacent files used to develop JupyterLab.

The initial work would move WebSocket/REST wrapper code around, and prepare for extensibility:

• DocumentConnectionManager
  • add registerConnector, probably with a key (e.g. package name) and rank
  • remove all WebSocket-related code with façades for the methods provided by the connectors
• LanguageServerManager
  • remove all REST code with façades for the methods provided by the connectors
• SocketConnector
  • re-add the WebSocket and REST code here
  • provide this as in new plugin that requires the main plugin

With this functionality extracted, a second connector would be added that supported the postMessage protocol, including WebWorker, iframe, and other browser tabs. This level of isolation (vs running in the main UI loop) is critical, as the Language Server architecture does not have very pretty failure modes, and many widely-used servers will simply die and need to be restarted. Further, outside of pure JS servers, a number of Language Servers can be compiled to browser-executable WebAssembly, but often at a significant, unpredictable runtime cost.

• WorkerConnector
  • add an implementation that manages one-server-session-per-postMessage-compatible containers
  • allow for registering new sources of workers
  • use schema-constrained postMessage to communicate between the main UI loop and a worker
    • LSP messages
    • contents
    • settings
    • stderr

In parallel, but likely in a @jupyter-lsp repo, a concrete implementation of a WebWorker-based JSON server would demonstrate this functionality.

Additional context

The initial goal is to make additive, backwards-compatible changes such as to not break compatibility with downstream packages, most notably those published as open source by Jupyter Code of Conduct-adhering @jupyter-lsp subproject. Other, uninspectable, proprietary, and variously-well-behaved downstreams certainly exist, however, so this means leaving the public and protected method signatures basically unchanged.

Below are some sketches of how the architecture would evolve:

As Is

classDiagram 
    class ILSPDocumentConnectionManager {
        connected: ISignal~IDocumentConnectionData~
        disconnected: ISignal~IDocumentConnectionData~
        connect(): Promise~ILSPConnection~
        disconnect(TLanguageServerId)
    }

    ILSPDocumentConnectionManager --o ILanguageServerManager : languageServerManager
    class ILanguageServerManager {
        sessions: TSessionMap;
        sessionsChanged: ISignal~void~;
    }

    lsp-plugin -- ILSPDocumentConnectionManager : provides
    class lsp-plugin["🔌 @jupyterlab/lsp-extension:plugin"] { }

To Be

classDiagram 
    class ILSPConnector {
        connected: Signal~IDocumentConnectionData~
        disconnected: Signal~IDocumentConnectionData~
        connect(): Promise~ILSPConnection~
        connections: Map~URI, ILSPConnection~
        disconnect(TLanguageServerId)
    }

    ILSPDocumentConnectionManager --o ILanguageServerManager : languageServerManager
    ILanguageServerManager --|> ILSPSessionProvider : extends
    class ILanguageServerManager { }

    class ILSPSessionProvider {
        sessions: TSessionMap;
        sessionsChanged: ISignal~void~;
    }
    
    ILSPDocumentConnectionManager --|> ILSPConnector : extends
    class ILSPDocumentConnectionManager { 
        registerConnector(int, ILSPConnector): void
    }

    ILSPSocketConnector --|> ILSPConnector : extends
    ILSPSocketConnector --|> ILSPSessionProvider : extends
    class ILSPSocketConnector["🆕 ILSPSocketConnector"] { }
    
    ILSPWorkerConnector --|> ILSPConnector : extends
    ILSPWorkerConnector --|> ILSPSessionProvider : extends
    class ILSPWorkerConnector["🆕 ILSPWorkerConnector"] { 
        addFactory(IWorkerLSOptions) 
    }

    lsp-plugin -- ILSPDocumentConnectionManager : provides
    class lsp-plugin["🔌 @jupyterlab/lsp-extension:plugin"] { }

    lsp-socket-plugin ..> ILSPDocumentConnectionManager : requires
    lsp-socket-plugin -- ILSPSocketConnector : provides
    class lsp-socket-plugin["🆕🔌 @jupyterlab/lsp-socket:plugin"] { }
    
    lsp-worker-plugin ..> ILSPDocumentConnectionManager : requires
    lsp-worker-plugin -- ILSPWorkerConnector : provides
    class lsp-worker-plugin["🆕🔌 @jupyterlab/lsp-worker:plugin"] { }

    lsp-worker-json-plugin ..> ILSPWorkerConnector : requires
    class lsp-worker-json-plugin["🆕🔌 @jupyter-lsp/lsp-worker-json:plugin"] { }

Out of Scope

With the second connector out of the way, any number of other implementations would be possible, without impairing the ability of the architecture and user experience to be further evolved.

KernelConnector

Co-locating a kernel and a language server has a number of advantages and challenges, but is too divisive to implement here.

An early prototype (predating jupyterlab-lsp) showed hand-made LSP messages passed over the high-level Jupyter Widget transport, running in-loop with ipykernel/IPython.

Another prototype PR to jupyter-lsp demonstrated a proxy kernel, which reused the entire python-based LanguageServerManager to manage multiple language servers, communicating with the frontend over a well-characterized Jupyter kernel WebSocket, using the comm protocol.

Other, proprietary implementations have been described in the weekly Jupyter Server call for entirely separate mechanisms.

Multiple Connections

Many languages would benefit from the insights provided by multiple simultaneous Language Servers. This is orthogonal to the proposed work, as it would almost certainly entail breaking changes.

[bollwyvl]

Folk that had previously expressed opinions about this topic:

• @krassowski
• @SylvainCorlay
• @jasongrout
• @jtpio