VSCode Semantic Tokens Sample Extension Does Not Work

[colin-grant-work]

Feature Description:

On current master (29ebe4c), it appears that the VSCode semantic tokens sample extension does not work. It appears to activate successfully, but the code to provideDocumentSemanticTokens is never called. This is the case in all themes, even with the preference editor.SemanticHighlightingEnabled set to true.

VSCode:

Theia:

OS: RHEL7
Platform: Electron
Version: 29ebe4c

[colin-grant-work]

After fixing a bug with the prefrence setting (#10511) and setting editor.semanticHighlighting.enabled to true, I can get to this screenshot:

There seem to be two problems remaining:

• Theia's builtin themes do not have semanticHighlighting set to true, so the 'configuredByTheme' setting mostly equates to false, unlike in VSCode.
• In VSCode, if you force semantic highlighting on a theme that doesn't support it, it appears that the editor is able to fall back to colors provided by the grammar for tokens whose semantic specification it doesn't understand. Theia fails and falls back to the default color (plain white, e.g.).

[colin-grant-work]

In VSCode, if you force semantic highlighting on a theme that doesn't support it, it appears that the editor is able to fall back to colors provided by the grammar for tokens whose semantic specification it doesn't understand. Theia fails and falls back to the default color (plain white, e.g.).

This seems to be a consequence of the way we consume Monaco. We use the IStandaloneThemeService(Impl):

theia/packages/monaco/src/typings/monaco/index.d.ts

Lines 937 to 954 in d149f51

	export interface IStandaloneThemeService extends monaco.theme.IThemeService {
	// https://github.com/theia-ide/vscode/blob/standalone/0.23.x/src/vs/editor/standalone/browser/standaloneThemeServiceImpl.ts#L200
	readonly _knownThemes: Map<string, IStandaloneTheme>;
	setTheme(themeName: string): void;
	setAutoDetectHighContrast(autoDetectHighContrast: boolean): void;
	defineTheme(themeName: string, themeData: monaco.editor.IStandaloneThemeData): void;
	getColorTheme(): IStandaloneTheme;
	setColorMapOverride(colorMapOverride: monaco.color.Color[] | null): void;
	}
	export class StandaloneThemeServiceImpl implements IStandaloneThemeService {
	readonly _knownThemes: Map<string, IStandaloneTheme>;
	setTheme(themeName: string): void { }
	setAutoDetectHighContrast(autoDetectHighContrast: boolean): void { }
	defineTheme(themeName: string, themeData: monaco.editor.IStandaloneThemeData): void { }
	getColorTheme(): IStandaloneTheme { }
	setColorMapOverride(colorMapOverride: monaco.color.Color[] | null): void { }
	}

Which implements a getTokenStyleMetadata() method:

https://github.com/microsoft/vscode/blob/224701f74d0f275bbe814f3689729ec323499478/src/vs/editor/standalone/browser/standaloneThemeServiceImpl.ts#L157-L169

that bottoms out here and returns the default editor colors, with no reference to the grammar:

https://github.com/microsoft/vscode/blob/224701f74d0f275bbe814f3689729ec323499478/src/vs/editor/common/modes/supports/tokenization.ts#L377

Internally, VSCode uses the ColorThemeData#getTokenStyleMetadata method:

https://github.com/microsoft/vscode/blob/224701f74d0f275bbe814f3689729ec323499478/src/vs/workbench/services/themes/common/colorThemeData.ts#L264-L277

Which bottoms out in a check for what the normal grammar color would have been:

https://github.com/microsoft/vscode/blob/224701f74d0f275bbe814f3689729ec323499478/src/vs/workbench/services/themes/common/colorThemeData.ts#L198-L217

Since we don't have access to that service in the current set up of our monaco package, we're missing that fallback.

[colin-grant-work]

Theia's builtin themes do not have semanticHighlighting set to true, so the 'configuredByTheme' setting mostly equates to false, unlike in VSCode.

The StandaloneTheme implementation used by Theia's monaco integration sets all theme's semanticHighlighting to false, so updating the themes has no effect on that behavior.