x/tools/gopls: semantic tokenizing of string escape/format characters

[soypat]

What version of Go are you using (go version)?

$ gopls version
golang.org/x/tools/gopls v0.6.10
    golang.org/x/tools/gopls@v0.6.10 h1:8Ebz8PymS2umcuCFhoz67unyJfWsipjTIrkBUF9kypg=

$ VsCode settings:
 "gopls": {	
        "ui.semanticTokens": true,
	},
    "editor.semanticHighlighting.enabled": true,

What did you do?

Write following code in VSCode with semanticTokens enabled

const httpResponse = "HTTP/1.0 200 OK\r\nContent-Type: text/html\r\nPragma: no-cache\r\n\r\n<h2>..::TinyGo Rocks::..</h2>"
fmt.Sprintf("%v", httpResponse)

What did you expect to see?

Semantic tokenization of \n\r\t escaped characters and format characters like %v in strings inside a Format function.

What did you see instead?

Monocolor string like the one github displays.

[pjweinb]

What should semantic tokens return in this case? (the possibilities are in the section on semantic tokens in https://microsoft.github.io/language-server-protocol/specifications/specification-3-17/)

[soypat]

Maybe an unused token, such as class could be used for such end. I'm not sure how to check what VScode uses for other languages, is there a way?

From here the following semantic tokens are available:

Semantic Tokens

• type
• class
• enum
• struct
• typeParameter
• parameter
• variable
• property
• enumMember
• event
• function
• method
• macro
• keyword
• modifier
• comment
• string
• number
• regexp
• operator

Semantic Token Modifiers

• declaration
• definition
• readonly
• static
• deprecated
• abstract
• async
• modification
• documentation
• defaultLibrary

[soypat]

Looks like the default coloring for these characters before enabling semantic tokenizing is property.

[pjweinb]

This is more a vscode issue than a gopls issue, but here's the vscode story. Based on experiments I believe what happens is the following, to decide the foreground color:

1. If there is a semantic token, use that color.
2. If not, if there is a textmate label, use that color.
3. Otherwise, use the default color.

In this case the semantic token is a string, so vscode uses the color for strings. If gopls didn't return string semantic tokens, then vscode would show the colors from the textmate grammar, which would show different colors for \n etc. In semantic tokens I know of no way to get different colors in the string (i tried all 10 of the SemanticTokenModifiers, and they all use the same string color.)

So there are two possiblities for gopls:

1. Leave things the way they are. Then the vscode user's choice would be semantic tokens and drab strings, or turn off semantic tokens and get colorful strings.
2. Don't return string semantic tokens. Then vscode users would get colorful strings, but other LSP clients would get incomplete semantic tokens. I do not know if there are any LSP clients using semantic tokens, other than vscode.

[pjweinb]

See the comment in #45753.

There are only two choices, as I see it:

1. Leave things the way they are. Then the vscode user's choice would be semantic tokens and drab numbers, or turn off semantic tokens and get colorful numbers.
2. Don't return number semantic tokens. Then vscode users would get colorful numbers, but other LSP clients would get incomplete semantic tokens. I do not know if there are any LSP clients using semantic tokens, other than vscode.

[soypat]

See the comment in #45753.

There are only two choices, as I see it:

1. Leave things the way they are. Then the vscode user's choice would be semantic tokens and drab numbers, or turn off semantic tokens and get colorful numbers.

2. Don't return number semantic tokens. Then vscode users would get colorful numbers, but other LSP clients would get incomplete semantic tokens. I do not know if there are any LSP clients using semantic tokens, other than vscode.

@pjweinb I think this comment was maybe meant for #45792?

[pjweinb]

just substitute 'string' for 'number'. The choices are the same.

…

On Wed, Apr 28, 2021 at 4:26 PM Patricio Whittingslow < ***@***.***> wrote: See the comment in #45753 <#45753>. There are only two choices, as I see it: 1. Leave things the way they are. Then the vscode user's choice would be semantic tokens and drab numbers, or turn off semantic tokens and get colorful numbers. 2. Don't return number semantic tokens. Then vscode users would get colorful numbers, but other LSP clients would get incomplete semantic tokens. I do not know if there are any LSP clients using semantic tokens, other than vscode. I think this comment was maybe meant for #45792 <#45792>? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#45753 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABJIAI5YSHRM74W7VUKZIUDTLBVP7ANCNFSM43QPKBLQ> .

[soypat]

Well I don't believe it is reasonable to be picky about what is implemented from the LSP. Option 1 seems like the better of two options. That said, is there any chance this could be brought in as an optional parameter to gopls? Something that can be modified in settings.json in VSCode for example.