Support for semantic syntax highlighting

[ghost]

(This is as much a feature request both for the server itself and the VSCode extension.)

I'm currently using texlab with a customised syntax highlighting extension for VSCode (https://marketplace.visualstudio.com/items?itemName=vomout.latex-syntax) to have special highlighting for \todo and \footnote. I also miss the texlab extension not giving any specific meaning and colours to \label, \cite, \cref, \eqref, etc. I'm happy for this to be a separate package—I like how lean the texlab vscode package is very lean whereas my custom (LaTeX-Workshop derived) grammar is ugly and bloated. It nevertheless does the job.

However, there appears to be a much better way. While basic TextMate grammars cannot easily tag syntax highlighting scopes with arbitrary function names (this is what my extension pretty much does for specific commands: adds footnote or todo to the end of the scope name—of the command and its parameters), it seems that language servers can more easily provide rich semantic highlighting information. See https://code.visualstudio.com/api/language-extensions/semantic-highlight-guide and the sample language at https://github.com/Microsoft/vscode-extension-samples/tree/master/semantic-tokens-sample. So I think it would be feasible for the texlab language server to pretty much tag the scope of every command (and its parameters) with the specific name of that command. In that way user customisations could easily highlight specific commands (including highlighting author initial-prefixed custom \XYtodo with author-specific colours). (Unfortunately, it does not seem possible for extensions to contribute additional syntax highlighting colours: microsoft/vscode#97753.)

What do you think?

[ghost]

I was told there's some LSP-relevant discussion in microsoft/language-server-protocol#18.

[pfoerster]

What do you think?

I think semantic highlighting is a great idea. In addition to that, the new editor.semanticTokenColorCustomizations setting seems pretty interesting.
We will make a patch release soon, which fixes some bugs. Afterwards, you can expect a 3.0.0 release with semantic highlighting.

[clason]

@pfoerster Now that Neovim has support for semantic tokens, I suddenly care about this :)

I remember we discussed this previously and agreed that it should be used sparingly -- if my TeX file ends up looking like a clown car crashed into a paint store window, that is... less than helpful.

So it should not try to replicate syntax highlighting (that's what tree-sitter-latex is for) but only show things that (only!) texlab knows about. It's also probably better to add these one at a time when someone comes up with a useful idea, instead of trying to COLOR ALL THE THINGS from the get-go.

With this in mind, here's some ideas that I think could be useful (to varying degree):

1. mark commands and environments that are defined in the current project (as opposed to builtin or from a package);
2. mark labels and citation keys that are not defined in the current project (this would be "diagnostics light", but having that low-level information without running a compilation first would be quite useful);
3. mark labels that are not used in the current project (so I can toggle the equation number off, or see at a glance that I have used equation* with a referenced equation);
4. mark labels according to environment (math-like, theorem-like, float-like);
5. mark reference keys according to bibtype (article, book, misc, ...)
6. mark plain TeX commands like $$ as deprecated :)

Of these, I could see myself relying on 2 and 3, possibly also 5. And 6 could be very pedagogical ;)

The standard token names are a poor fit for this, of course, but the spec allows defining new ones, or texlab could just re-purpose them, similar to what it does with completionItemKinds and symbolKinds.

[pfoerster]

@clason
Thanks for the awesome list of suggestions!

I have created a draft PR (#869), which implements (2) and (3) for labels as a starting point. Do you mind trying it out? I think, most of the other suggestions can be easily implemented, too.