Draft: Implement a language server

[fwcd]

This branch contains some experimental work on implementing the proposed language server. In particular, it uses the lsp library and adds a new mode that can be invoked by running

koka --language-server

Additionally, the VSCode extension now supports talking to the language server, provided that the appropriate paths have been configured (this is done through a VSCode setting, koka.languageServer.command and koka.languageServer.enabled).

Implemented LSP features:

• Hover (using the provided range map)
• Diagnostics (currently only after save)
• Go-to-Definition (top-level constants/constructors/types only)
• Document Symbols (aka. Outline)
• Code Completion (not context-aware yet)
• Workspace Symbols
• Signature Help
  ...

Screenshots:

See more

Development

To set up VSCode for language server development, do the following:

• Build the Koka compiler using stack build
• Navigate into <repo>/support/vscode/koka.language-koka and run:
  • npm install to install the dependencies
  • npm run build to compile the extension (or, alternatively, npm run watch to continuously recompile it in the background)
• Create a launch configuration in <repo>/.vscode/launch.json with the following:

{
    "version": "0.2.0",
    "configurations": [
        {
            "args": [
                "--extensionDevelopmentPath=${workspaceFolder}/support/vscode/koka.language-koka"
            ],
            "name": "Launch Extension",
            "outFiles": [
                "${workspaceFolder}/support/vscode/koka.language-koka/out/**/*.js"
            ],
            "request": "launch",
            "type": "pwa-extensionHost"
        }
    ]
}

• Now you should be able to run the extension in VSCode's debug panel:

• In the newly launched VSCode instance, make sure to add the following to your settings.json:

"koka.languageServer.enabled": true,
"koka.languageServer.command": "stack exec -- koka --language-server",
"koka.languageServer.cwd": "<path to koka repo>"

• Opening any Koka file should now trigger the language server
  • You may have to reload your VSCode instance after changing the settings

[daanx]

Wow @fwcd -- this is great! Super happy to see this I will check it out soon and play with it.

One request, can you base it of the latest dev branch -- this actually has a bunch of recent commits that improves the rangemap for things with effects and handlers. (If it is troublesome to rebase, dont worry about it).

Also, I see you moved to GHC 8.8 -- we can do that but I am a bit worried about pulling in new dependencies on dependent type extensions; can we work around that? Similarly for the "lens" library? (I would like to limit dependencies on intricate Haskell features as much as possible with an eye towards a future Koka version)

Thanks again!

[daanx]

Just to add: note in the documentation on https://koka-lang.github.io/koka/doc/book.html you can see the tooltips when hovering over checked code (with a green border): these tooltips are generated by the src/static/Colorize.hs module from a rangeMap; perhaps you can reuse this for the hover in VS? (not sure)

[fwcd]

Hi @daanx, sure, I will rebase the branch!

The LSP library unfortunately requires GHC 8.8 due to haskell/lsp#263 and pulls in these dependent type extensions (which had to be mentioned explicitly here, due to not being available on Stackage yet). Also, the lenses are mostly just a convenience, so not really needed, though the LSP library uses them internally if I recall correctly.

Perhaps building the language server as a separate executable would be a better option? That would keep these dependencies more cleanly decoupled from the core compiler, though I am not sure how well that would fit into the current architecture.

Generating hovers using the existing colorization seems like a great idea, I will have a look at that. :)

[fwcd]

No problem, I have merged the dev branch and added some instructions on how to set up the VSCode extension for language server development (in the PR description). Additionally, basic code completion should now work.

[daanx]

Super! It is amazing that you were able to create this in less than a week! :-)

I just tried it out -- it build correctly but then the npm build goes wrong; I am guessing perhaps the npm install has the wrong version or perhaps lacks some packages? Can you help me with this -- I do not know enough yet to fix it.

It looks like this:

c:\Users\daan\dev\koka-lsp\support\vscode\koka.language-koka>npm install
npm notice created a lockfile as package-lock.json. You should commit this file.
added 8 packages from 47 contributors and audited 8 packages in 2.025s
found 0 vulnerabilities


c:\Users\daan\dev\koka-lsp\support\vscode\koka.language-koka>npm run build

> language-koka@2.0.0 build c:\Users\daan\dev\koka-lsp\support\vscode\koka.language-koka
> tsc -p .

node_modules/vscode-languageclient/lib/client.d.ts:2:19 - error TS2305: Module '"../../vscode-languageserver-protocol/lib/common/api"' has no exported member 'RPCMessageType'.

2 import { Message, RPCMessageType, ResponseError, RequestType, RequestType0, RequestHandler, RequestHandler0, GenericRequestHandler, NotificationType, NotificationType0, NotificationHandler, NotificationHandler0, GenericNotificationHandler, MessageReader, MessageWriter, Trace, Event, ClientCapabilities, TextDocumentRegistrationOptions, InitializeParams, InitializeResult, InitializeError, ServerCapabilities, DocumentSelector, DidOpenTextDocumentNotification, DidChangeTextDocumentNotification, DidCloseTextDocumentNotification, DidSaveTextDocumentNotification, WillSaveTextDocumentNotification, WillSaveTextDocumentWaitUntilRequest, FileEvent, CompletionRequest, HoverRequest, SignatureHelpRequest, DefinitionRequest, ReferencesRequest, DocumentHighlightRequest, WorkspaceSymbolRequest, CodeActionRequest, DocumentFormattingRequest, DocumentRangeFormattingRequest, DocumentOnTypeFormattingRequest, RenameRequest, DocumentLinkRequest, ProgressType, StaticRegistrationOptions, DocumentColorRequest, DeclarationRequest, FoldingRangeRequest, ImplementationRequest, SelectionRangeRequest, TypeDefinitionRequest, Proposed } from 'vscode-languageserver-protocol';
                    ~~~~~~~~~~~~~~

node_modules/vscode-languageclient/lib/client.d.ts:413:33 - error TS2314: Generic type 'RequestType0<R, E>' requires 2 type argument(s).

413     sendRequest<R, E, RO>(type: RequestType0<R, E, RO>, token?: CancellationToken): Promise<R>;
                                    ~~~~~~~~~~~~~~~~~~~~~~

node_modules/vscode-languageclient/lib/client.d.ts:414:36 - error TS2314: Generic type 'RequestType<P, R, E>' requires 3 type argument(s).

414     sendRequest<P, R, E, RO>(type: RequestType<P, R, E, RO>, params: P, token?: CancellationToken): Promise<R>;
                                       ~~~~~~~~~~~~~~~~~~~~~~~~

node_modules/vscode-languageclient/lib/client.d.ts:417:31 - error TS2314: Generic type 'RequestType0<R, E>' requires 2 type argument(s).

417     onRequest<R, E, RO>(type: RequestType0<R, E, RO>, handler: RequestHandler0<R, E>): void;
                                  ~~~~~~~~~~~~~~~~~~~~~~

node_modules/vscode-languageclient/lib/client.d.ts:418:34 - error TS2314: Generic type 'RequestType<P, R, E>' requires 3 type argument(s).

418     onRequest<P, R, E, RO>(type: RequestType<P, R, E, RO>, handler: RequestHandler<P, R, E>): void;
                                     ~~~~~~~~~~~~~~~~~~~~~~~~

node_modules/vscode-languageclient/lib/client.d.ts:420:32 - error TS2315: Type 'NotificationType0' is not generic.

420     sendNotification<RO>(type: NotificationType0<RO>): void;
                                   ~~~~~~~~~~~~~~~~~~~~~

node_modules/vscode-languageclient/lib/client.d.ts:421:35 - error TS2314: Generic type 'NotificationType<P>' requires 1 type argument(s).

421     sendNotification<P, RO>(type: NotificationType<P, RO>, params?: P): void;
                                      ~~~~~~~~~~~~~~~~~~~~~~~

node_modules/vscode-languageclient/lib/client.d.ts:424:30 - error TS2315: Type 'NotificationType0' is not generic.

424     onNotification<RO>(type: NotificationType0<RO>, handler: NotificationHandler0): void;
                                 ~~~~~~~~~~~~~~~~~~~~~

node_modules/vscode-languageclient/lib/client.d.ts:425:33 - error TS2314: Generic type 'NotificationType<P>' requires 1 type argument(s).

425     onNotification<P, RO>(type: NotificationType<P, RO>, handler: NotificationHandler<P>): void;
                                    ~~~~~~~~~~~~~~~~~~~~~~~

node_modules/vscode-languageclient/lib/client.d.ts:498:32 - error TS2708: Cannot use namespace 'Proposed' as a value.

498     getFeature(request: typeof Proposed.CallHierarchyPrepareRequest.method): DynamicFeature<TextDocumentRegistrationOptions> & TextDocumentProviderFeature<TypeDefinitionProvider>;
                                   ~~~~~~~~

node_modules/vscode-languageclient/lib/workspaceFolders.d.ts:3:48 - error TS2305: Module '"../../vscode-languageserver-protocol/lib/common/api"' has no exported member 'RPCMessageType'.

3 import { ClientCapabilities, InitializeParams, RPCMessageType, ServerCapabilities, WorkspaceFoldersRequest } from 'vscode-languageserver-protocol';
                                                 ~~~~~~~~~~~~~~


Found 11 errors.

npm ERR! code ELIFECYCLE
npm ERR! errno 2
npm ERR! language-koka@2.0.0 build: `tsc -p .`
npm ERR! Exit status 2
npm ERR!
npm ERR! Failed at the language-koka@2.0.0 build script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     C:\Users\daan\AppData\Roaming\npm-cache\_logs\2020-12-14T23_22_12_878Z-debug.log

[daanx]

More generally, how does this work? I am surprised we still need typescript? I thought it was all Haskell with a single executable.

I saw it needed a ton more libraries etc than plain Koka -- I think it is indeed best to make this a separate executable; what do you think is the best way forward? I thought:

• make two separate branches: main-lsp and dev-lsp that add the extra dependencies and that build two executables, the plain compiler and the language server .
• when creating a binary installer, I would use the main-lsp branch and also install the needed language server files (so I will need to understand too which files are needed to make it run

Anyway, cannot wait to get this running :-) Will it work on macOS as well?

[daanx]

Ooh, it just built.. :-)

[fwcd]

Ah, there seemed to be some inconsistency with the declared VSCode dependency. Re-running npm install and npm run build should now work.

The TypeScript/JavaScript part is a little bit of glue code that tells VSCode where to find and how to run the language server executable. Everything else is implemented in Haskell though. The entire language server source code in located in src/LanguageServer (apart from the --language-server flag), so it should be easy to work on it separately.

Also, sure, it runs on macOS and Linux just fine (I've tested it on both). :)

[daanx]

It works! Amazing! I need to play with it more but this is incredible. Very cool!
I will try to understand more how it works and perhaps help out :-)

Some quick question:

• If I start a function call with ( it does not show the parameter types or info. That is difficult to implement I guess?
• the types for the current module are fully prefixed with the path, like c:/Users/daan/dev/koka-lsp/samples/basic/rbtree/bench : (n : int) -> <console,div> () -- is this normal? I guess the koka "root" names are not yet set up right. Koka usually finds the shortest path from any of the "roots" (in the --include path) and takes that as the canonical name)

Very fun -- thanks for starting this project!

[fwcd]

Signature help (also context-aware completion) is a bit more difficult to implement than regular completion as it involves working with the 'broken' AST as the user types. I haven't looked into it that much yet, but that would definitely be nice to have.

Ideally, the language server should recompile on every keystroke, possibly with a short debouncer/delay, so it can show errors and warnings in real-time. This would, however, require passing the source text directly to the compiler rather than just the file paths, since the file contents might not have been saved to disk (The lsp-library manages an in-memory VFS of the user's project files which is synchronized with the editor's text buffer for this purpose).

[fwcd]

thanks for starting this project!

Both the language and the compiler are very fun to experiment with, so thank you for all the work that went into it! Also, the approach of declaring effects rather than using monad transformers seems really interesting to me. :)

[timjs]

What's the status of this? Would be nice to have a LSP for Koka!

[TimWhiting]

@fwcd

Take a look at #394. We are planning on merging soon. It was based originally off of your work & we've added you to the copyright notices in the files you were involved in.

[fwcd]

Thanks, cool work! Really happy to see this being picked up again. It has been a while since I've worked on this, so I am afraid I can't comment much on the code, but that sounds pretty good.