x/tools: Language Server Protocol Support

[cueckoo]

Originally opened by @rudolph9 in cuelang/cue#142

Here is an example of a YAML file LSP https://github.com/redhat-developer/yaml-language-server

I've never created one so I can't say for certian how much work it would be but it would be great for integrations in to vim, emacs, VSCode, etc.

https://langserver.org/

[cueckoo]

Original reply by @rudolph9 in cuelang/cue#142 (comment)

Somewhat related, looks like someone made a vim plugin https://github.com/jjo/vim-cue

[cueckoo]

Original reply by @jlongtine in cuelang/cue#142 (comment)

Some other links as well:

• https://github.com/sourcegraph/go-lsp
• https://github.com/sourcegraph/go-langserver

I was chatting with @mpvl about this a bit at KubeCon this week, and I think this could be highly valuable for CUE.

[cueckoo]

Original reply by @mpvl in cuelang/cue#142 (comment)

To get a better idea of what this should look like: what kind of functionality and analysis would people want to see supported by a LSP implementation?

[cueckoo]

Original reply by @mxey in cuelang/cue#142 (comment)

To get a better idea of what this should look like: what kind of functionality and analysis would people want to see supported by a LSP implementation?

• Syntax checking/validation (basically, show errors without having to run cue eval first)
• Symbols for completion, documentation and jumping to definition

basically, I'd like the same editing experience for Kubernetes objects in CUE as you can get for YAML in VS Code

[cueckoo]

Original reply by @mpvl in cuelang/cue#142 (comment)

To get a better idea of what this should look like: what kind of functionality and analysis would people want to see supported by a LSP implementation?

• find all locations contributing to a current position
• expand implied fields into current struct, to allow editing them as if it were a form
• grey out or otherwise mark fields that could be removed with trim (highly important fields)
• remove unchanged implied fields (implied fields could be marked with @Implied()
• trim at cursor
• hoist/relax (anti-unify) at cursor

[cueckoo]

Original reply by @rudolph9 in cuelang/cue#142 (comment)

Long term goal, I'd like to build something like Barlin.

This gets into some very meta concepts underpinning cue but theoretically a cue evaluator could be written in cue similar to the way Barlin utilizes a scheme interpreter implemented in MiniKanren which implemented scheme 🤔

[cueckoo]

Original reply by @zchee in cuelang/cue#142 (comment)

I created packages that lsp for Go. (for my project, Neovim LSP plugin written in Go :D)
It almost works IIRC. I'm using zchee/nvim-lsp (private).

• https://github.com/go-language-server/protocol

If we develop lsp for CUE written in Go, it might be helpful that package.

[cueckoo]

Original reply by @myitcv in cuelang/cue#142 (comment)

We can likely reuse some of the work in golang.org/x/tools/internal/lsp/... to get us started.

Probably worth starting with simple diagnostics. That will allow us to flesh out a basic skeleton for what the LSP server should look like (including how to handle modules, a la Go modules).

AFAIK, syntax definitions fall outside the spec of LSP. I plan to implement a Vim plugin in Go using govim for AST driven syntax highlighting to try out the parser/AST.

[cueckoo]

Original reply by @metalmatze in cuelang/cue#142 (comment)

I've talked to @mpvl at FOSDEM and it might be worth checking out the Prometheus Language Server @slrtbtfs created as part of his internship in our team. It reuses parts of gopls but in a more general purpose way. @slrtbtfs should have more details about why things needed to be changed, in case you have questions. That should already get this going a bit quicker :)

[cueckoo]

Original reply by @slrtbtfs in cuelang/cue#142 (comment)

Hi,

I've had a quick look at cue, and it should probably be possible to reuse parts of the PromQL language server for the cue language server.

What is definitely reusable and turned out be very useful is the golang.org/x/tools/internal/lsp/proctocol package, which contains (almost) all the types defined in the LSP specification.

What could be useful for you, too, is forking the github.com/prometheus-community/promql-langserver/langserver/cache package, removing all the PromQL stuff there and replacing the yaml parser with the cue parser. This package basically takes care of all the statekeeping in the language server.

Then the only thing left to do is implementing the protocol.Server interface. You can start with leaving everything unimplemented and then add more and more language features over time.

Feel free to reach out if you have any questions.

[cueckoo]

Original reply by @mikelnrd in cuelang/cue#142 (comment)

Another more recent development related to language servers is The Language Server Index Format (LSIF).

Basically the idea is to spit out the analysis to a JSON file. Sourcegraph (a code search tool) lets you upload the LSIF file and gives you language-server-like code navigation (but without the language server even running!).

Perhaps cue could spit out the LSIF JSON data?

I've just grep'ed for my notes on the topic and thought I'd share them here in case they are helpful. See below.

Loving cue at the moment and the vscode-cue plugin. Can't wait for a language server!

..............................................................
https://code.visualstudio.com/blogs/2019/02/19/lsif

https://github.com/microsoft/lsif-node/blob/master/README.md
https://github.com/Microsoft/language-server-protocol/blob/master/indexFormat/specification.md

vscode extension (to serve LSP from the file)
https://github.com/microsoft/lsif-node/blob/master/README.md#lsif-extension

https://about.sourcegraph.com/blog/sourcegraph-3.9
https://about.sourcegraph.com/blog/writing-an-lsif-indexer
https://about.sourcegraph.com/blog/sourcegraph-3.8
https://about.sourcegraph.com/blog/code-intelligence-with-lsif

https://docs.sourcegraph.com/user/code_intelligence/lsif

go get -u github.com/sourcegraph/lsif-go/cmd/lsif-go

https://github.com/sourcegraph/lsif-go
https://www.youtube.com/watch?v=fMIRKRj_A88
https://youtu.be/fMIRKRj_A88?t=218

https://github.com/microsoft/lsif-node/tree/master/tsc

lsif-protocol: Protocol defined as TypeScript interfaces
lsif-util: Utility tools for LSIF development
lsif-tsc: LSIF indexer for TypeScript
lsif-npm: Linker for NPM monikers

[cueckoo]

Original reply by @galli-leo in cuelang/cue#142 (comment)

Hi,

I semi-hacked together a working implementation (using promql as a base) over the past two days. You can see it running in action in vscode below :). Currently, I have the following working:

• Diagnostics: parse errors, compile errors and even output of cue vet
• Definition: navigate to definition (including multiple, see GIF)
• Hovering: showing definition on hover + comments, works decentish, but could use some work
• Completion: Extremely basic at the moment, would need a lot more work here

(There might be one or two crashes though 😅 )

Should I fork this project and add what I have up until know, then open a PR?
Or should I just commit it to a personal repo, that's separate?

Some thoughts I had during my implementation, most of these are probably my fault for not being familiar with the codebase though 😛 :

• It is very annoying that compilation fails immediately, if there is any error (or maybe I haven't found the correct options yet?). At the moment I need to keep the last Instance of a document around, otherwise I only have the AST. Seems to be possible by using build.Instance instead. However, if a top level identifier is unknown, inst.Value() will still return nothing, so would need to access some internal stuff.
• Navigating the AST and Instance is very annoying. I wrote my own Cursor for the AST, but nothing yet for Instance.
• Matching between something in the AST and a corresponding Value is even more annoying, that code is very messy and should probably be reworked. Currently, I iterate through all Values in the Instance and check if val.Source() == node (node is an ast.Node).
• There doesn't seem to be a way to get a Value corresponding to a field definition, which made some stuff a lot harder.
• Mapping between source locations is also not trivial and seems to be the cause of one of the crashes.
• There doesn't seem to be a distinction between an ast.Ident when used e.g. a: b (i.e. one is field definition label, the other field definition type?). Ideally, there was a way to know which one is which, since certain stuff (like auto complete or hover), have different info based on that (at least the way I implemented it at the moment).

Lastly, I wanted to say, that I really love this project! I think it's really cool and we plan on using it in our new infrastructure for all config management (hence why I decided to spend some time implementing this :)).

[cueckoo]

Original reply by @verdverm in cuelang/cue#142 (comment)

@galli-leo this is awesome!

Have you signed the Google CLA? If not, we can get you a link to the instructions

Cue accepts PRs from here or Gerrit, with some preference for Gerrit as it is the actual source of truth for the code. Gerrit uses a slightly different commit methodology which is pretty cool to experience if you are up for it. Depending on which method you prefer, we'll follow up with more details.

Are you aware the syntax is changing slightly? In particular, I see your example (really cool by the way) has the old style definitions with <ident> :: _. The newer syntax is #<ident>: _ and list comprehension is a little different too. I'm sure there are some others I am missing.

There are some other changes that may impact the lang-server implementation, and hopefully they will only make it easier. As a general FYI, Cue is going through some bigger changes right now on its way to stability later this year.

Have you joined the slack group? (link to join is here: https://cuelang.org/community)

[cueckoo]

Original reply by @galli-leo in cuelang/cue#142 (comment)

@verdverm

Have you signed the Google CLA? If not, we can get you a link to the instructions

Yep signed it already.

Cue accepts PRs from here or Gerrit, with some preference for Gerrit as it is the actual source of truth for the code. Gerrit uses a slightly different commit methodology which is pretty cool to experience if you are up for it. Depending on which method you prefer, we'll follow up with more details.

I prefer Github, since I don't have experience with gerrit. But I am also fine with using gerrit, if that makes it easier for you :)

Are you aware the syntax is changing slightly? In particular, I see your example (really cool by the way) has the old style definitions with :: _. The newer syntax is #: _ and list comprehension is a little different too. I'm sure there are some others I am missing.

I am now ;) To be honest, I delved into this without having familiarized myself 100%!((MISSING)probably not even 80%!:(MISSING)P) with the language spec (in hindsight maybe not the best idea).

There are some other changes that may impact the lang-server implementation, and hopefully they will only make it easier. As a general FYI, Cue is going through some bigger changes right now on its way to stability later this year.

I saw the issues regarding that. Is there a central document detailing those changes?

Have you joined the slack group?

Yep! Will ask further questions there.

[cueckoo]

Original reply by @buremba in cuelang/cue#142 (comment)

@galli-leo looks great! Any plans to open-source the LSP?

[cueckoo]

Original reply by @myitcv in cuelang/cue#142 (comment)

LSP is close to the top of the list to be worked on in the near future... so this will get some attention soon.

[cueckoo]

Original reply by @alex-de-oliveira in cuelang/cue#142 (comment)

This would be absolutely invaluable

[cueckoo]

Original reply by @uhthomas in cuelang/cue#142 (comment)

I haven't seen any mention of Sublime Text 4 yet, so I'll share some quick thoughts.

Sublime Text 4 has a generic LSP implementation which works pretty well. I've been using it in conjunction with gopls for about a year now and both have improved significantly in that time.

It's an exciting prospect to have a similar LSP for CUE.

[cueckoo]

Original reply by @eadlam in cuelang/cue#142 (comment)

@myitcv I'm interested in helping with the LSP

[cueckoo]

Original reply by @myitcv in cuelang/cue#142 (comment)

I am now starting work on a first cut of cuepls. This work will complement the proposal for CUE modules which will be published this week.

This first cut of cuepls should hopefully support the work @jansorg and others have done in #250 on IntelliJ support.

Once we have the right "structure" in place (cuepls will live in the main CUE repo), I will message back here and create issues for aspects/features of cuepls that remain to be implemented. That way, people who would like to contribute can coordinate via an issue so that we don't have a race condition on people working on the same feature. We can also flesh out design issues, discuss whether changes to core CUE packages are required etc.

I will now tag all LSP-related issues with cuepls; adding help wanted and good first issue where appropriate.

That said, any contribution will of course be greatly appreciated! Because when we have an initial implementation of cuepls, actually using it and reporting bugs will be a massive help in and of itself.

[cueckoo]

Original reply by @samschlegel in cuelang/cue#142 (comment)

Also interested with helping with this as this is the main thing blocking our adoption of CUE, and I really want to adopt it 😄

[cueckoo]

Original reply by @eadlam in cuelang/cue#142 (comment)

This is also the main thing blocking CUE adoption at my org.

[cueckoo]

Original reply by @leoluk in cuelang/cue#142 (comment)

Note that while there's no LSP yet, there are a number of language/syntax plugins for various editors:

• VSCode
• IntelliJ
• Vim

...and probably more. That might be good enough to unblock adoption while waiting for full autocompletion :)

The IntellIj one in particular is commercially supported by us and has a full parser/lexer and will highlight syntax errors.

[cueckoo]

Original reply by @myitcv in cuelang/cue#142 (comment)

Thanks for everyone's patience here. This is in progress; we very much understand the importance of good editor support for the success of CUE.

[cueckoo]

Original reply by @samschlegel in cuelang/cue#142 (comment)

Let me know if there's any way I could help!

Note that while there's no LSP yet, there are a number of language/syntax plugins for various editors:

Yeah I've been using those for some things I've been trying out, but in order to kick off replacing all of our Jsonnet we're going to need things like go to definition, type information, and autocomplete for both VSCode and IntelliJ