Go highlighted differently after January 2024 update

[mfridman]

Does this issue occur when all extensions are disabled?: Yes/No

Version: 1.86.0
Commit: 0504748
Date: 2024-01-31T10:29:11.933Z
Electron: 27.2.3
ElectronBuildId: 26495564
Chromium: 118.0.5993.159
Node.js: 18.17.1
V8: 11.8.172.18-electron.0
OS: Darwin arm64 23.2.0

Steps to Reproduce:

1. Updated VS Code to the latest January 2024 release
2. A previously working theme, Atom One Dark Theme, is broken following this update. Most of the colors for identifiers are different and very noticeable.

[mfridman]

I downgraded to 1.85.2 and it resolved my problem. I didn't make any other changes outside bumping versions (in both directions).

Note, to prevent VS Code from updating itself had to add "update.mode": "none" to the settings.json file.

---

Here's a before and after just as an example

Before

After

[AbyssWatcher]

I'm running into the same issue. The syntax highlighting specifically for Go code is noticeably different than before. This is with all installed extensions disabled.

Similarly downgrading to 1.85.2 also resolved my problem.

---

Before

Version: 1.85.2
Commit: 8b37750
Date: 2024-01-18T06:40:11.430Z
Electron: 25.9.7
ElectronBuildId: 26354273
Chromium: 114.0.5735.289
Node.js: 18.15.0
V8: 11.4.183.29-electron.0
OS: Darwin arm64 23.2.0

After

Version: 1.86.0
Commit: 0504748
Date: 2024-01-31T10:29:11.933Z
Electron: 27.2.3
ElectronBuildId: 26495564
Chromium: 118.0.5993.159
Node.js: 18.17.1
V8: 11.8.172.18-electron.0
OS: Darwin arm64 23.2.0

[aeschli]

I suspect this is related to the update of the go grammar

[alexr00]

Yes, this is related to the update of the go grammar. As far as I can tell, all the color changes are improvements: tokens that weren't colored before are now colored. I should have included that we have a new Go grammar in the release notes though.

[alexr00]

Added to release notes with microsoft/vscode-docs#7012

[mfridman]

That's a bit unfortunate, because this new grammar is (subjectively) worse as compared to whatever was there previously.

For those that preferred the existing grammar, are there any potential solutions?

Could you point me to whatever was there before? Also, is there a settings to override the default?

[alexr00]

No setting. Can you describe the ways in which it is worse? The old grammar left a lot of tokens that should be highlighted un-highlighted. It also wasn't being actively maintained. The new one not only colors more tokens, but it also fixes bugs that existed in the old one.

[mfridman]

Ye sorry, I should have rephrased, the grammar is probably better, but the way it alters the theme is worse.

Highlighting more tokens is not always desired. So I guess it was a happy accident the previous grammar and theme worked the way they did.

I guess the only real solution is to come up with a custom theme to match existing behaviour or find a new one.

ps. After at least 6 yr of Go/VSCode almost daily I hope you can understand how jarring this might be. It's a bit silly, but it becomes like muscle memory in a way.

[alexr00]

After at least 6 yr of Go/VSCode almost daily I hope you can understand how jarring this might be. It's a bit silly, but it becomes like muscle memory in a way.

@mfridman this is fair, and thanks for being a dedicated user of VS Code! One thing you can always do is selectively change the color of some of the tokens using the editor.tokenColorCustomizations setting. This will let you keep the benefit of the updated Go grammar, but let you have full control over what each token looks like. Maybe there are some new colors that you want to keep, and this approach will also let you do that. For example:

NOTE: for users of other theme than the One Dark Pro theme, you may want to change "foreground": "#ABB2BF" to be #fff or whatever the default color is for your theme. I chose #ABB2BF specifically for One Dark Pro.

"editor.tokenColorCustomizations": {
		"[*Light*]": { // can change this to be the full name of your theme
			"textMateRules": [
				{
					"scope": [
						"variable.other.property.go",
						"variable.other.go", // add more scopes here if you care about light themes as you find them
					],
					"settings": {
						"foreground": "#000"
					}
				}
			]
		},
		"[*Dark*]": { // can change this to be the full name of your theme
			"textMateRules": [
				{
					"scope": [
						"variable.other.property.go",
						"variable.other.go", // add more scopes here if you care about dark themes as you find them
					],
					"settings": {
						"foreground": "#ABB2BF"
					}
				}
			]
		}
	},

Will look like

To find the scopes to add to the above setting, just open a Go file with highlighting you want to change, run the command "Developer: Inspect Editor Tokens and Scopes", then put your cursor in the token that you don't like. The "textmate scopes" will be what you need to copy:

An aside: I use a very similar theme, and it highlights a lot of tokens in red. I can see why that would be particularly jarring if you are not used to seeing it.

[mfridman]

Thank you for the detailed response, I didn't know those knobs existed in the settings and they are more than sufficient to tweak based on personal preference.

If you code long enough you either see yourself dropping syntax highlighting altogether or maintaining your own theme. Hehe, I hope not.

[adsr303]

I think what could help here is if there was a distinction between variables and fields of those variables in the new syntax.

[zackslash]

I understand that making more tokens available and that the upgrade is beneficial in that sense. However, wouldn't it be logical to keep the original behavior for these new tokens? At the moment it appears to be a step back in terms of user-friendliness.

[starball5]

Related on Stack Overflow: VS Code 1.86 update changed Go syntax highlighting. Why?

[adsr303]

Related on Stack Overflow: VS Code 1.86 update changed Go syntax highlighting. Why?

Interesting that both the maintainer of the Go Syntax extension and the VS Code team were aware of the visual changes, as there are comparison screenshots on the extension's front page, and neither of them concluded that the new syntax highlighting would be too aggressive. That would make for an interesting UX study.

[alexr00]

I would actually say that any tokens that are unhighlighted is a bug/deficiency in the grammar. For example, in typescript there are no unhighlighted tokens:

[zackslash]

I would actually say that any tokens that are unhighlighted is a bug/deficiency in the grammar. For example, in typescript there are no unhighlighted tokens:

In my opinion, the original behavior could arguably be viewed as a deficiency. While I believe modifying it is justifiable, I think a more effective approach would be to provide existing users with an upgrade path (or some sort of option to maintain the existing behaviour given it's been present over many years) instead of imposing what could be considered quite an impacful change on them.

[alexr00]

I think a more effective approach would be to provide existing users with an upgrade path (or some sort of option to maintain the existing behavior given it's been present over many years)

I'm sad to say that we don't have a good, existing, way of doing this. For the built in themes, we could have held all users back on the old behavior of not coloring token, but we couldn't have only done it for existing users. For non-build-in themes, we would not be able to anything.

[zackslash]

I think a more effective approach would be to provide existing users with an upgrade path (or some sort of option to maintain the existing behavior given it's been present over many years)

I'm sad to say that we don't have a good, existing, way of doing this. For the built in themes, we could have held all users back on the old behavior of not coloring token, but we couldn't have only done it for existing users. For non-build-in themes, we would not be able to anything.

Historically other IDEs making similar changes to syntax colors have provided a set of 'legacy' themes to maintain the pre-existing behaviour that users could optionally change to. In any case, it seems too late at this point but I would like to offer this as feedback for future changes.

[RedCMD]

Could you point me to whatever was there before? Also, is there a settings to override the default?

@mfridman https://marketplace.visualstudio.com/items?itemName=jeff-hykin.better-go-syntax
https://github.com/jeff-hykin/better-go-syntax

'legacy' themes

syntax highlighters don't modify themes

if you want to make a bug report, here's the new repo https://github.com/worlpaker/go-syntax

[realvasko]

One thing that seems to not be working with the new grammar though is format verbs? Old grammar would highlight %s and %v differently, which was neat. According to the example screenshots under https://github.com/worlpaker/go-syntax it should be working though.

Does anyone else experience this problem? I have this in my settings.json:

  "gopls": {
    "ui.semanticTokens": true
  }

Trying to run Developer: Inspect Editor Tokens ans Scopes just returns the string info.

[odoland]

Regex seems to break the formatting. Issue created here:

worlpaker/go-syntax#6

[alexr00]

@realvasko I'm seeing those tokens get highlighted differently:

Is this what you expect?

[alexr00]

@odoland per the response in worlpaker/go-syntax#6 this issue has been fixed in a newer version of the Go grammar. The fix has already been pulled into VS Code insiders. If you want to try out VS Code insiders it installs side by side with stable and, if you're willing to help, it would be great to have some Go users who will notice grammar changes using VS Code insiders to help us catch issues like this before we release right now as we have newly adopted the grammar at https://github.com/worlpaker/go-syntax.

There is always a bit of turbulence when we adopt a new grammar as the grammar suddenly has a much wider reach than it used to. The upside is that the owner of the grammar we've adopted is willing to address issues, including longstanding issues that existed in the previous grammar.

[jrefior]

I'm having this issue as well. I can't find a single color scheme where Go code looks like the screenshots that advertise the color scheme, or looks like how the color scheme applies to other languages. I've been using Synthwave '84 for years, and now my JavaScript code still looks like SynthWave '84, but my Go code looks like something different. The same issue seems to be there for other color schemes: the color rendering for Go code just doesn't look like what the theme designer intended.

Achieving a better grammar sounds like great progress! But I wonder if there might be room for improvement on how the various tokens match up with existing color scheme/theme designs, in ways that are similar to how themes apply to other languages?

For example, here's what Synthwave '84 is supposed to look like according to the Visual Studio Marketplace page:

Here's what JavaScript looks like on my system with the theme installed (looks similar to the theme advertisement):

Go code on my system looked about like that too, until recently. Now with the vscode changes, it looks like this:

It's hard to even recognize that as being the same color scheme.

[hueuebi]

After at least 6 yr of Go/VSCode almost daily I hope you can understand how jarring this might be. It's a bit silly, but it becomes like muscle memory in a way.

@mfridman this is fair, and thanks for being a dedicated user of VS Code! One thing you can always do is selectively change the color of some of the tokens using the editor.tokenColorCustomizations setting. This will let you keep the benefit of the updated Go grammar, but let you have full control over what each token looks like. Maybe there are some new colors that you want to keep, and this approach will also let you do that. For example:

NOTE: for users of other theme than the One Dark Pro theme, you may want to change "foreground": "#ABB2BF" to be #fff or whatever the default color is for your theme. I chose #ABB2BF specifically for One Dark Pro.

"editor.tokenColorCustomizations": {
		"[*Light*]": { // can change this to be the full name of your theme
			"textMateRules": [
				{
					"scope": [
						"variable.other.property.go",
						"variable.other.go", // add more scopes here if you care about light themes as you find them
					],
					"settings": {
						"foreground": "#000"
					}
				}
			]
		},
		"[*Dark*]": { // can change this to be the full name of your theme
			"textMateRules": [
				{
					"scope": [
						"variable.other.property.go",
						"variable.other.go", // add more scopes here if you care about dark themes as you find them
					],
					"settings": {
						"foreground": "#ABB2BF"
					}
				}
			]
		}
	},

Will look like

To find the scopes to add to the above setting, just open a Go file with highlighting you want to change, run the command "Developer: Inspect Editor Tokens and Scopes", then put your cursor in the token that you don't like. The "textmate scopes" will be what you need to copy:

An aside: I use a very similar theme, and it highlights a lot of tokens in red. I can see why that would be particularly jarring if you are not used to seeing it.

Sadly this doesn't work the moment you turn on
"ui.semanticTokens" = true