LSP: semantic tokens support

[jdrouhard]

This is a continuation of #15723. This will take into account all the feedback accumulated over the year(s) there and take advantage of newer core LSP functionality to hopefully provide a good baseline for semantic tokens support.

• Determine if core should provide basic highlighting support
• Fix document version synchronization
• Fix documentation to match new module and API
• Update tests for new functionality and API

[jdrouhard]

@mfussenegger @gpanders @clason

I think this is ready for an initial review. I've been using this branch pretty extensively the past few days and it feels pretty solid. I'll begin on tests, but just wanted to start soliciting feedback on overall design and API stuff before I get too far testing something that may change.

[jdrouhard]

@mfussenegger - thanks for taking the time to help with the API design and for looking things over.

@bfredl - I'd like to tag you for some input here as you have significant involvement with highlighting in general.

General question - should I be using nvim_buf_add_highlight(), or should I use the extmark API directly? The former doesn't let me specify a priority for these highlights, but I believe it's a slightly faster code path.

[clason]

General question - should I be using nvim_buf_add_highlight(), or should I use the extmark API directly? The former doesn't let me specify a priority for these highlights, but I believe it's a slightly faster code path.

extmarks, hands down. It should follow (and integrate with) treesitter highlighting as closely as possible. In particular, we need proper priority support.

[clason]

I really want to keep modifiers out of this PR, as this feature has been derailed often enough.

When this is merged, we'll open a dedicated issue to discuss how these should behave based on 1. how modifiers are actually used in practice (e.g., what are the "relevant" combinations?) and 2. how this fits into the basic highlighting design in Neovim. I cannot stress enough that we do not want to add a special-case code path for these few highlights if it's at all possible to fit this into a general (improved) design.