[LSP] Support semantic tokens feature

[ericdallo]

I'd like to suggest adding support for LSP semantic tokens :)

What is it?

LSP semantic tokens is a feature introduced on latest spec release 3.16 and clojure-lsp is one of the few servers that support it. It was called semantic highlighting in the past and basically needs the server to return tokens, saying which they are, like functions, enums, namespace, variable, keyword, etc and the client color it on the document following user theme, giving to the user a better and improved highlighting on their code :)

Do I need it for Clojure?

Here is a comparation of it on emacs with cider(REPL) connected:

Remember that this would make highlight works without the need of a REPL ;)

This is a screenshot of Calva without semantic tokens as of now with default theme:

This is a screenshot of Calva without semantic tokens as of now with monokai theme:

Emacs(lsp-mode) with semantic tokens enabled:

You can see how tokens like @db/db, string/upper-case, and especially for Clojure, macros, are highlighted with a specific color, also some themes don't handle well clojure keywords and things like that, which with LSP probably would fix it

What Calva needs to do?

On server side, this feature is enabled by default already (:semantic-tokens?) as it exists for months already and it works great IMO.
On client side, it needs to send during initialization that it supports the semantic tokens. The good news are, AFAICS there is no need for Calva to configure anything as the vscode-languageclient already handles that, but I could not make it work for calva for some reason and I suspect that is related to the version of that lib, maybe upgrading to 7.0.0 just like Dart-Code, which supports semantic-tokens as well, did?

I think this would be a nice addition for Clojure users, LMK what you think :)
c/c @PEZ @bpringe

[bpringe]

Yeah, I think we would need to upgrade the language client library. The version we use now is using 3.15.3 of the protocol.

As far as using this feature, I'll defer to @PEZ to see what he thinks.

It's great that you added that feature to the server!

[PEZ]

I think it is great! We can let the users have a setting for it, with enabled as default, imo. Can it be toggled on/off while the server is running?

[ericdallo]

@PEZ, ATM no, but I think it could be done as LSP has a specific method for things like that, didConfiguratiinChanged I think

[ericdallo]

@PEZ It's possible via https://microsoft.github.io/language-server-protocol/specification#client_registerCapability.
This is not blocker for Calva though.
I'll try to implement on clojure-lsp soon