x/tools/gopls: semantic tokenizing of numeric modifiers and imaginary unit

[soypat]

What version of Go are you using (go version)?

$ gopls version
golang.org/x/tools/gopls v0.6.10
    golang.org/x/tools/gopls@v0.6.10 h1:8Ebz8PymS2umcuCFhoz67unyJfWsipjTIrkBUF9kypg=

$ VsCode settings:
 "gopls": {	
        "ui.semanticTokens": true,
	},
    "editor.semanticHighlighting.enabled": true,

What did you do?

Write following code in VSCode with semanticTokens enabled

const (
    hexReg  = 0x1337
    bitmask = 0b0110_1100

    google = 1E100
    planck = 2e-23
    // How to tokenize this?
    c = 1+20i
)

What did you expect to see?

Semantic tokenization of 0x, 0b, E, and e numeric modifiers. This was colored before enabling LSP semantic tokenization. The coloring for these could be keyword (this was the default coloring I was accustomed to seeing before enabling semantic tokenizing). As a bonus maybe the 0 in octal notation could be colored for clarity?

I'm not sure how i (imaginary unit) should be tokenized. It seems to serve a different purpose as it is a number by itself. It would probably be helpful to color it though! Maybe property token for it?

What did you see instead?

Monocolor numbers.

[stamblerre]

/cc @pjweinb

[ALTree]

0x1337, 1e100 and 1+20i all are a single number, all should be one single token, and if you have colourization enabled I would not expect e.g. the e in 1e100 to be of a different colour (why would it be? It's part of the number and has no meaning by itself). Same for the + and the i in complex literals.

[soypat]

I could really do without the e and i colorization in 1e10i. That said, I would benefit from some sort of colorization of 0x and 0b in 0x1337 and 0b0101 since they have a fundamental impact to the interpretation of the digits. It would make surfing Register definitions, flags and bitmasks a more pleasant task.

[hyangah]

@pjweinb FYI I added the numeric number highlighting here jeff-hykin/better-go-syntax@43a6d98 in the textmate rule. I don't know if this makes sense for 'semantic token' feature - they are all numbers.

[hyangah]

0x, 0b, etc are not keywords strictly speaking.

Semantic highlighting and LSP spec allows custom token types/modifiers (https://code.visualstudio.com/api/language-extensions/semantic-highlight-guide#custom-token-types-and-modifiers). I wonder if we can at least differentiate hex/dec/binary numbers using modifiers.

[pjweinb]

i think the right way to do this is to use the new options 'noSemanticString' and 'noSemanticNumber' which disables semantic tokens for strings and numbers, allowing the native colorization to show through, (It's hard for users to get useful colors from the semantic token modifiers)

…

On Tue, Sep 6, 2022 at 9:02 AM Hyang-Ah Hana Kim ***@***.***> wrote: 0x, 0b, etc are not keywords strictly speaking. Semantic highlighting and LSP spec allows custom token types/modifiers ( https://code.visualstudio.com/api/language-extensions/semantic-highlight-guide#custom-token-types-and-modifiers). I wonder if we can at least differentiate hex/dec/binary numbers using modifiers. — Reply to this email directly, view it on GitHub <#45792 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABJIAI3XS5JDGUEZWRJUDX3V446FRANCNFSM43T6WICA> . You are receiving this because you were assigned.Message ID: ***@***.***>

[hyangah]

Closing as the noSemanticNumber setting makes the syntax highlighting for numbers to be handled by the default editor syntax highlighting. Thanks @pjweinb