Setup with Jupyter LSP

[ikasou]

Hello,
I have tried to set this up to work in Jupyter Lab (using WolframLanguageForJupyter) with @krassowski/jupyterlab-lsp as follows:
Using macOS Big Sur 11.6.3, system python 3.8, and Jupyter core packages:
jupyter_client : 6.1.12
jupyter_core : 4.9.0
jupyter_server : 1.11.2
jupyterlab : 3.2.8
nbclient : 0.5.4
Installed Wolfram Engine 13 in /Applications
Cloned via 'git clone https://github.com/WolframResearch/WolframLanguageForJupyter.git' and
Built WolframLanguageForJupyter-0.9.2.paclet via ./configure-jupyter.wls build
Ran PacletInstall["WolframLanguageForJupyter-0.9.2.paclet"]
And ConfigureJupyter["Add", "JupyterInstallation" -> "/Users/ikasou/Library/Python/3.8/bin/jupyter"]
So far so good, notebooks in Jupyter work like a charm!

For the LSP setup, I am using

  "LanguageServerManager": {
    "language_servers": {
      "wolfram-lspserver": {
        "version": 2,
        "argv": [
		      "/Applications/Wolfram Engine.app/Contents/MacOS/WolframKernel",
		      "-noinit",
          	      "-noprompt",
		      "-nopaclet",
		      "-noicon",
		      "-nostartuppaclets",
		      "-run",
		      "Needs[\"LSPServer`\"];LSPServer`StartServer[\"/Users/ikasou/Library/Logs/WolframLSP\"]"
	],
        "languages": ["vnd.wolfram.m", "Wolfram Language"],
        "mime_types": ["text/x-vnd.wolfram.m", "application/x-vnd.wolfram.m"],
        "display_name": "Wolfram Language"
      }
    }
  }
}

The system seems to think that the Server initialises properly:

However looking at the logs I see that the server crashes immediately as soon as it starts:
kernelLog-2022-02-01_10-54-51.txt

The key lines seem to be:

10:54:52.182 Starting server... (If this is the last line you see, then StartServer[] may have been called in an unexpected way and the server is hanging.)
10:54:52.182 



KeyExistsQ::invrl: The argument Null is not a valid Association or a list of rules.

KeyExistsQ::invrl: The argument Null is not a valid Association or a list of rules.

KeyExistsQ::invrl: The argument Null is not a valid Association or a list of rules.

KeyExistsQ::invrl: The argument Null is not a valid Association or a list of rules.

KeyExistsQ::invrl: The argument Missing[KeyAbsent, codeAction] is not a valid Association or a list of rules.
10:54:53.169 


10:54:53.169 invalid contents result (should match {_?AssociationQ ...}): handleContent[<|jsonrpc -> 2.0, method -> workspace/didChangeConfiguration, params -> <|settings -> <||>|>|>]
10:54:53.169 

10:54:53.170 Language Server kernel did not shutdown properly.

So it appears it is not getting the format it is expecting from jupyter-lsp somehow. Looking at the Jupyter logs I see (I have removed irrelevant lines):

[I 2022-02-01 10:54:32.883 ServerApp] jupyter_lsp | extension was successfully linked.
[I 2022-02-01 10:54:32.891 ServerApp] jupyter_server_mathjax | extension was successfully linked.
[I 2022-02-01 10:54:32.905 ServerApp] jupyterlab | extension was successfully linked.
[I 2022-02-01 10:54:32.905 ServerApp] jupyterlab_git | extension was successfully linked.
[I 2022-02-01 10:54:33.135 ServerApp] nbclassic | extension was successfully linked.
[I 2022-02-01 10:54:33.135 ServerApp] nbdime | extension was successfully linked.
[I 2022-02-01 10:54:33.202 ServerApp] nbclassic | extension was successfully loaded.
[I 2022-02-01 10:54:33.452 ServerApp] Skipped non-installed server(s): bash-language-server, dockerfile-language-server-nodejs, javascript-typescript-langserver, julia-language-server, pyright, sql-language-server, texlab, typescript-language-server, unified-language-server, vscode-css-languageserver-bin, vscode-html-languageserver-bin, vscode-json-languageserver-bin, yaml-language-server
[I 2022-02-01 10:54:33.616 ServerApp] jupyter_lsp | extension was successfully loaded.
[I 2022-02-01 10:54:33.617 ServerApp] jupyter_server_mathjax | extension was successfully loaded.
[I 2022-02-01 10:54:33.619 LabApp] JupyterLab extension loaded from /Users/ikasou/Library/Python/3.8/lib/python/site-packages/jupyterlab
[I 2022-02-01 10:54:33.619 LabApp] JupyterLab application directory is /Users/ikasou/Library/Python/3.8/share/jupyter/lab
[I 2022-02-01 10:54:33.622 ServerApp] jupyterlab | extension was successfully loaded.
[I 2022-02-01 10:54:33.627 ServerApp] jupyterlab_git | extension was successfully loaded.
[I 2022-02-01 10:54:33.705 ServerApp] nbdime | extension was successfully loaded.
[I 2022-02-01 10:54:33.707 ServerApp] Serving notebooks from local directory: /Users/ikasou/Documents
[I 2022-02-01 10:54:33.707 ServerApp] Jupyter Server 1.11.2 is running at:
[I 2022-02-01 10:54:38.830 LabApp] Build is up to date
[W 2022-02-01 10:54:40.579 ServerApp] Notebook Untitled.ipynb is not trusted
[I 2022-02-01 10:54:41.185 ServerApp] Kernel started: a397aea0-fdec-4e24-9142-4acc13b5c592
[I 2022-02-01 10:54:41.215 ServerApp] Kernel started: bf112415-60cd-4aab-aeda-7442fe93f56f
[I 2022-02-01 10:54:48.141 ServerApp] Adapting from protocol version 5.3.0 (kernel bf112415-60cd-4aab-aeda-7442fe93f56f) to 5.3 (client).
10:54:52.178 $CommandLine: {/Applications/Wolfram Engine.app/Contents/MacOS/WolframKernel, -noinit, -noprompt, -nopaclet, -noicon, -nostartuppaclets, -run, Needs["LSPServer`"];LSPServer`StartServer["/Users/ikasou/Library/Logs/WolframLSP"]}
10:54:52.179 


10:54:52.179 $commProcess: StdIO
10:54:52.180 


10:54:52.180 $ProcessID: 3724
10:54:52.180 


10:54:52.181 $ParentProcessID: 3687
10:54:52.181 


10:54:52.182 Starting server... (If this is the last line you see, then StartServer[] may have been called in an unexpected way and the server is hanging.)
10:54:52.182 



KeyExistsQ::invrl: The argument Null is not a valid Association or a list of rules.

KeyExistsQ::invrl: The argument Null is not a valid Association or a list of rules.

KeyExistsQ::invrl: The argument Null is not a valid Association or a list of rules.

KeyExistsQ::invrl: The argument Null is not a valid Association or a list of rules.

KeyExistsQ::invrl: The argument Missing[KeyAbsent, codeAction] is not a valid Association or a list of rules.
10:54:53.169 


10:54:53.169 invalid contents result (should match {_?AssociationQ ...}): handleContent[<|jsonrpc -> 2.0, method -> workspace/didChangeConfiguration, params -> <|settings -> <||>|>|>]
10:54:53.169 


10:54:53.170 Language Server kernel did not shutdown properly.
10:54:53.171 
10:54:53.172 This is the command that was used:
10:54:53.172 {/Applications/Wolfram Engine.app/Contents/MacOS/WolframKernel, -noinit, -noprompt, -nopaclet, -noicon, -nostartuppaclets, -run, Needs["LSPServer`"];LSPServer`StartServer["/Users/ikasou/Library/Logs/WolframLSP"]}
10:54:53.173 
10:54:53.173 To help diagnose the problem, run this in a notebook:
             Needs["LSPServer`"]
             LSPServer`RunServerDiagnostic[{"/Applications/Wolfram Engine.app/Contents/MacOS/WolframKernel", "-noinit", "-noprompt", "-nopaclet", "-noicon", "-nostartuppaclets", "-run", "Needs[\"LSPServer`\"];LSPServer`StartServer[\"/Users/ikasou/Library/Logs/WolframLSP\"]"}]
10:54:53.174 
10:54:53.174 Fix any problems then restart and try again.
10:54:53.175 


10:54:53.175 KERNEL IS EXITING HARD
10:54:53.176 


[E 2022-02-01 10:56:40.808 ServerApp] <LspStdIoWriter(parent=<LanguageServerSession(language_server=wolfram-lspserver, argv=['/Applications/Wolfram Engine.app/Contents/MacOS/WolframKernel', '-noinit', '-noprompt', '-nopaclet', '-noicon', '-nostartuppaclets', '-run', 'Needs["LSPServer`"];LSPServer`StartServer["/Users/ikasou/Library/Logs/WolframLSP"]'])>)> couldn't write message: Content-Length: 227
    
    {"jsonrpc":"2.0","method":"textDocument/didSave","params":{"textDocument":{"uri":"file:///Users/ikasou/Documents/.virtual_documents/Python/BerkeleyDRL_fall2021/hw1/cs285/Untitled1.ipynb","version":2},"text":"Range[3]\n\n\n\n"}}
    Traceback (most recent call last):
      File "/Users/ikasou/Library/Python/3.8/lib/python/site-packages/jupyter_lsp/stdio.py", line 193, in write
        await convert_yielded(self._write_one(response.encode("utf-8")))
      File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/concurrent/futures/thread.py", line 57, in run
        result = self.fn(*self.args, **self.kwargs)
      File "/Users/ikasou/Library/Python/3.8/lib/python/site-packages/jupyter_lsp/stdio.py", line 201, in _write_one
        self.stream.write(message)
    BrokenPipeError: [Errno 32] Broken pipe
[I 2022-02-01 11:21:50.158 ServerApp] Starting buffering for a397aea0-fdec-4e24-9142-4acc13b5c592:76e701b9-6579-457d-a533-e0a39947d0f2
[E 2022-02-01 11:21:59.101 ServerApp] <LspStdIoWriter(parent=<LanguageServerSession(language_server=wolfram-lspserver, argv=['/Applications/Wolfram Engine.app/Contents/MacOS/WolframKernel', '-noinit', '-noprompt', '-nopaclet', '-noicon', '-nostartuppaclets', '-run', 'Needs["LSPServer`"];LSPServer`StartServer["/Users/ikasou/Library/Logs/WolframLSP"]'])>)> couldn't write message: Content-Length: 230
    
    {"jsonrpc":"2.0","id":1,"method":"textDocument/hover","params":{"textDocument":{"uri":"file:///Users/ikasou/Documents/.virtual_documents/Python/BerkeleyDRL_fall2021/hw1/cs285/Untitled1.ipynb"},"position":{"line":0,"character":4}}}
    Traceback (most recent call last):
      File "/Users/ikasou/Library/Python/3.8/lib/python/site-packages/jupyter_lsp/stdio.py", line 193, in write
        await convert_yielded(self._write_one(response.encode("utf-8")))
      File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/concurrent/futures/thread.py", line 57, in run
        result = self.fn(*self.args, **self.kwargs)
      File "/Users/ikasou/Library/Python/3.8/lib/python/site-packages/jupyter_lsp/stdio.py", line 201, in _write_one
        self.stream.write(message)
    BrokenPipeError: [Errno 32] Broken pipe

I can even run the diagnostics suggestion from the error message in the notebook:

Needs["LSPServer`"]
LSPServer`RunServerDiagnostic[{"/Applications/Wolfram Engine.app/Contents/MacOS/WolframKernel", "-noinit", "-noprompt", "-nopaclet", "-noicon", "-nostartuppaclets", "-run", "Needs[\"LSPServer`\"];LSPServer`StartServer[\"/Users/ikasou/Library/Logs/WolframLSP\"]"}]

But it doesn't shed any new light, besides some versioning info:

Running Language Server diagnostic...

Kernel that is running RunServerDiagnostic[] ($CommandLine[[1]]): 
 
>   /Applications/Wolfram Engine.app/Contents/Resources/Wolfram\
 
>    Player.app/Contents/MacOS/WolframKernel
Kernel that RunServerDiagnostic[] will start (RunServerDiagnostic[{kernel,\
 
>    ...}]): /Applications/Wolfram Engine.app/Contents/MacOS/WolframKernel
WARNING: RunServerDiagnostic[] should be run with same kernel that\
 
>   RunServerDiagnostic[] will start.

Starting Language Server kernel with command: 
 
>   {/Applications/Wolfram Engine.app/Contents/MacOS/WolframKernel, -noinit, 
 
>    -noprompt, -nopaclet, -noicon, -nostartuppaclets, -run, 
 
>    Needs["LSPServer`"];LSPServer`StartServer["/Users/ikasou/Library/Logs/Wo\
 
>     lframLSP"]}

If any messages are printed below, they must be fixed.

Writing initialize...
initialize was successful.
Writing diagnostics...
INFORMATION: Kernel version: 13.
INFORMATION: LSPServer version: 1.4
INFORMATION: CodeParser version: 1.4
INFORMATION: CodeInspector version: 1.4
INFORMATION: CodeFormatter version: 1.4
INFORMATION: LSPServer build date: Thu 09 Dec 2021 04:58:36
INFORMATION: CodeParser build date: Wed 08 Dec 2021 21:05:06
INFORMATION: CodeInspector build date: Wed 08 Dec 2021 18:22:29
INFORMATION: CodeFormatter build date: Wed 08 Dec 2021 21:19:45
diagnostics was successful.
Writing shutdown...
shutdown was successful.
Writing exit...
exit was successful.
INFO: Time to initialize server: 4.87181 seconds
No problems found.

I am more than happy to send any more logs that may be necessary, e.g. --debug level logs from Jupyter so we can see all the <LspStdioWriter()> message exchanges; however from memory they don't seem to shed much light as all json-rpc info is mostly opaque I think.

I appreciate this is work in progress but just thought I might give this a try anyway.

Thanks

[bostick]

This is probably a simple thing to fix. I will look at it.

[bostick]

There were a few edge cases exposed by Jupyter Lab LSP that were not being handled. I have fixed these up.

I was previously seeing the same problems you were, and now everything works for me.

[ikasou]

Thanks for taking a look. It initialises properly for me, too. Basic functionality seems to work: Typing over underlined keyword pops up a box with usage info. Tab completion of half-typed function names sadly doesn't work - any quick fixes for that?

[bostick]

Sorry, no quick fixes for completions. Adding support for completions is on the list of things to do though.