Write an autocompletion framework

[marijnh]

This involves a number of things that still have to be determined:

• What should completion sources look like (should be more flexible and less ad-hoc than the old system, support custom rendering of options, custom apply commands, sync and async operation)

• How does the UI plugin that handles completion get connected to the completion sources? Do sources act as separate plugins that expose services, or are they passed directly to the UI plugin?

• How do we structure a conventional completion UI (list below the cursor, type or arrow to select) in an accessible way?

[zikaari]

You might call it overkill, but perhaps could be a step in the right direction. I think there's a tonne we can learn from VSCode (monaco-editor). One of them is LSP (Language Server Protocol) specification. Which I think they defined and is now being used by few other tools.

LSP Specification | AutoCompletion

Now unlike their setup where Language "Server" runs as a separate process, CodeMirror can have WebWorkers if the user desires so^[1]. Editor changes can be synced with worker effciently using text-store which is cache based text container that does micro-edits (replace/insert/remove) efficiently.

[1] This is why I chose CodeMirror over monaco-editor. Latter comes with so much stuff built-in, that it uses has to use it own AMD loader in the page. Creating/Overriding global require variable. The worst part is that monaco-editor "manages" when to create/destroy language service WebWorkers thus purging the entire AST a worker might have created for itself.

Love CodeMirror for its modularity.

[marijnh]

Seeing whether it's feasible to communicate with language servers would be interesting—I expect there are some issues, like most of them not being written in JavaScript or easily runnable in a web worker, but maybe others have had the same problem and, I don't know, compiled such a server to WebAssembly or something.

Such a connection could act like just another source of completions that our completion UI could query.

[zikaari]

Exactly!

I think we shouldn't be concerned about how (typeof tech/backend) to write a language server/service. But more so being compliant with a standard as a client/frontend.

If we stick with that spec, a wrapper/shim around a language service will be only thing required to bridge to different technologies^[1], if they are tooooo far apart. Re-writing code just because API calls are different won't be necessary. ICompletionProvider will remain as ICompletionProvider and getCompletionItems will remain as such, so on and so forth.

[1] And that boils down to "modularity" of CodeMirror. Allowing users to plug some wires here and there and get things going, unlike being "locked-down" like monaco-editor.

[futurist]

For monaco users, It's excellent if can use existing Language syntax definition for getCompletionItems to extract syntax token and provide completions base on the context of input, or provide a way to convert the counterpart into CM syntax token completely (aka need little hack to do the translations)

https://microsoft.github.io/monaco-editor/monarch.html

[wylieconlon]

I've actually been working on supporting the full language server protocol in CodeMirror for the last month, and you can try many of the methods here: https://github.com/wylieconlon/lsp-editor-adapter

The biggest finding I've had is that Monaco does not fully use the Language Server Protocol when talking to a Service Worker, instead it's using what it calls a "language provider" which is an interface defined here: https://microsoft.github.io/monaco-editor/api/modules/monaco.languages.html

This interface is loosely inspired by the LSP, but is not the same. I can see CodeMirror using the same kind of interface which would allow the implementation of IDE features, regardless of whether they are implemented in-browser or on a server somewhere.

I also found that there are some undocumented requirements of the LSP, such as each document having a file:/// URI, which make it hard to translate to a browser context. For the project I linked above, I found that running on an actual server is a requirement.

[marijnh]

Nice work. Too bad that LSP isn't generic enough to fit a browser context without modification, but I guess that was to be expected.

[felipeochoa]

Here's some prior-art that might help for inspiration. Given Emacs has some similar constraints as a browser, could be a useful reference. It supports sync and async completions and is relatively modern (by Emacs standards)

https://github.com/company-mode/company-mode/wiki/Writing-backends

[curran]

What if the autocomplete feature was really simple and JS only? At extension construction time, the API could accept an async function that takes as input the partially completed word, and returns a promise that resolves to an array of strings (the autocomplete suggestions). The autocomplete extension could take care of rendering the selection list and implementing and suggestion selection interactions (arrow keys and enter I suppose).

Integration with various backends or standards seems out of scope for the first layer of an autocomplete framework. IMO the first layer should expose an extremely simple API, then other layers can be built on top of that. Thoughts?

[marijnh]

Having a standardized interface for completion sources, which multiple completion UIs can use, is definitely a good idea. It'd have to be a bit more complicated than word → word[], though — not all completions work on words, and many need further context (possibly the state of some on-line code analysis tool) to compute their result.

[curran]

What would it take to get something on the simpler side working, like Vim's autocomplete?

This is just word → word[], where the resulting words are words seen by the editor in the current session.

This feels like a subset of the functionality we'd ultimately want for the autocompletion framework, and may generate a useful strawman to start from, which can later be modified to handle various complexities and provide ways to hook into external code analysis tools.

[marijnh]

In the current CodeMirror, that (called 'anyword hint') is just a special case of the general highlighter, which scans the document for matching words.

So whichever way we'd go, we'd want to set up the functionality for collecting and displaying the hints in a generic way, so that you can plug in both something like this and a hyper-advanced code analysis engine.

[arguiot]

Having such UI would be great, but there should be an API to customize this UI. Like a callback being fired each time the suggestion list changes (basically each time an input is detected)

[firecrackerz]

only vscode-languageserver-protocol is enough, there is no need to implement vscode-ws-jsonrpc that part user can implement it by themselves.

[satishbabariya]

I would like codemirror.next to have pluggable minimal Auto-Complete Suggestions and languageserver-protocol as an extension.

[curran]

With the advancements of Lezer, whole new worlds of autocomplete opportunities may open up.

Really looking forward to seeing how this autocomplete framework turns out!

[marijnh]

This is in good shape now. I've opened #226 and #227 for open issues.