Join GitHub today
GitHub is home to over 40 million developers working together to host and review code, manage projects, and build software together.Sign up
This issue tracks the API proposal for semantic tokens.
In depth API explanation
A file can contain many tokens, perhaps even hundreds of thousands of tokens. Therefore, to improve the memory consumption around describing semantic tokens, we have decided to avoid allocating an object for each token and we represent tokens from a file as an array of integers. Furthermore, the position of each token is expressed relative to the token before it because most tokens remain stable relative to each other when edits are made in a file.
In short, each token takes 5 integers to represent, so a specific token
How to encode tokens
Here is an example for encoding a file with 3 tokens:
How tokens change when the document changes
Let's look at how tokens might change.
Continuing with the above example, suppose a new line was inserted at the top of the file. That would make all the tokens move down by one line (notice how the line has changed for each one):
The integer encoding of the tokens does not change substantially because of the delta-encoding of positions:
It is possible to express these new tokens in terms of an edit applied to the previous tokens:
Furthermore, let's assume that a new token has appeared on line 4:
The integer encoding of the tokens is:
Again, it is possible to express these new tokens in terms of an edit applied to the previous tokens:
When to return
Since microsoft/vscode-languageserver-node#367 proposed the protocol to be server pushing notification to the client, would it be fine for
In such case, the client is capable of requesting vscode to re-render the semantic highlighting once notification from server is received.
Am I correct that
So, for example, if I "goto definition" to the new file, the editor will first ask to only color the small visible part of the document (for faster response), and then it'll ask the second query to color the whole file (so that scrolling doesn't show boring colorless text)?
This is a super-important optimization,
Also I'd like to suggest that maybe, if we have ranges, we don't need
I don't like the presence of