Semantic Highlighting vs Injected Grammars

[mitsuhiko]

• VSCode Version: 1.52.1
• OS Version: Darwin x64 19.6.0

This is a meta bug report as I'm not sure if this feature is intentional or indeed a bug. Currently if one uses injectTo to "inject" a sub language into an already existing grammar this stops working when semantic highlighting is provided by a language server.

Configuration wise my extension does something like this:

    "grammars": [
      {
        "language": "insta-snapshots",
        "scopeName": "source.insta-snapshots",
        "path": "./syntaxes/insta-snapshots.tmLanguage.json"
      },
      {
        "scopeName": "source.inline-insta-snapshots",
        "injectTo": [
          "source.rust"
        ],
        "path": "./syntaxes/inline-insta-snapshots.tmLanguage.json",
        "embeddedLanguages": {
          "meta.embedded.inline-insta-snapshot": "insta-snapshots"
        }
      }
    ]

Then in the grammar I'm matching on something. This all works and eventually I end up injecting a custom sub syntax into strings that look like @r"""...""". The issue here now is that once the rust language server runs a "string" semantic token is put over the entire region which I just parsed which completely removes my custom syntax again.

The following screenshot shows the issue:

The token under the cursor is correctly determined to be keyword.insta but the styling in the theme is discarded because of the string semantic token which takes precedence.

Currently there does not appear to be a way to resolve this except for turning off semantic highlighting.

[vscodebot]

(Experimental duplicate detection)
Thanks for submitting this issue. Please also check if it is already covered by an existing one, like:

• Dynamic injection grammar contributions (#53885)

[alexdima]

This is intentional. TM tokens can only be an approximation of the parsed language/AST and cannot benefit from semantic analysis. That is why we have made the design decision that semantic tokens always overwrite TM tokens. The overwrite works like a mask, so if a range is not covered by a semantic token, the TM token is used instead.

For this particular case, I think you could try to coordinate with the rust extension / rust semantic tokens provider implementation to give you an option or an API that would prevent the semantic tokens provider from covering those strings. Also, if the semantic tokens provider just creates string tokens, (which would be equal with the tokens created by TM), then they could simply stop creating them altogether without any visible effects.

[mitsuhiko]

The issue I see with this is that injecting sub languages into strings is a very common thing. That applies to regular expressions in many languages, template engines in strings, embedded html and much more.

This right now means the moment semantic highlighting exists, all these use cases do not have a supported solution within vscode and require custom extension points in the relevant RLS.