Add .cpp at the end of the Textmate Colorization scopes.

[REYNEP]

Type: Feature Request

Currently none of the textmate Scopes [a.k.a Semantic token] have .cpp in them. Without that, no Theme can target Specific Language. And After using a lot of themes, it feels like, Most of the Theme Authors does that. Vanilla VSCode textMate scopes does have .cpp like language specifications at the end. And that helps a lot.... 🙂

[sean-mcmanus]

Are you referring to semantic token scopes (https://code.visualstudio.com/api/language-extensions/semantic-highlight-guide) or lexical/syntax scopes (https://code.visualstudio.com/api/language-extensions/syntax-highlight-guide)? Our extension doesn't provide any lexical scopes (they're obtained by VS Code from https://github.com/jeff-hykin/cpp-textmate-grammar/issues), and semantic tokens we provide are mapped to fallback textmate scopes by VS Code (https://code.visualstudio.com/api/language-extensions/semantic-highlight-guide#predefined-textmate-scope-mappings).

[Colengms]

@REYNEP When VS Code looks up a TextMate scope, it uses a fully qualified scope (including ".cpp" if the file is C++). I don't believe it's necessary for the C++ extension to specify this anywhere. There is a fallback approach that is used to match a scope in a Theme. It will try the fully qualified scope first, then reduce it until a match is found. For example: a.b.c.cpp is tried first, then a.b.c, a.b, then a.

That said, is there a specific scenario in which this lookup is not working as expected for you? Where exactly are you suggesting ".cpp" needs to be added?

Are you seeing an issue related to the TextMate fallback scopes used by our semantic token provider? Our semantic token provider supports both C and C++ sources, so cannot specify ".c" or ".cpp" in the fallback Textmate scopes for semantic tokens in semanticTokenScopes in our package.json. If VS Code is failing to add ".cpp" to those scopes when looking them up, that may be a VS Code issue.

[REYNEP]

@Colengms & Sean-mcmanus, I apologize for late reply....

I think even if a.b.c.cpp is tried, it still doesn't work, In details, even if .cpp in the end is available, still yet the SEMANTIC TOKEN SCOPE added by the Enhanced Colorization won't contain .cpp

One such case is:- variable.other.property [Added to member Variables of Struct/Class, I believe] even tho .cpp is available and Vanilla vsCode itself adds that.

Also couldn't C/C++ Extension get the fileName and then if .cpp/.cc/.cxx/.cp then add .cpp at all the Semantic Token Scopes?

[sean-mcmanus]

@Colengms The relevant doc appears to be https://code.visualstudio.com/api/language-extensions/semantic-highlight-guide#custom-textmate-scope-mappings. Should we add stuff like

      {
        "language": "cpp",
        "scopes": {
          "label": ["entity.name.label.cpp"]
        }
      },
      {
        "language": "c",
        "scopes": {
          "macro": ["entity.name.function.preprocessor.c"]
        }
      }

or is that the responsibility of the TextMate grammar provider (https://github.com/jeff-hykin/cpp-textmate-grammar/)?

[Colengms]

@REYNEP Some semantic fallback scopes are provided by VS Code itself, such as property -> variable.other.property (which you referred to). I would expect VS Code to internally append the current file's language ID when looking up styles for semantic token fallback scopes. If it does not, that would seem to be a bug in VS Code. I'd suggest opening an issue against VS Code directly, if you are not seeing it properly lookup a language-specific style for a variable.other.property resulting from a property semantic token (when a style is not available for the property semantic scope). I would expect VS Code should also apply the same behavior (adding language ID to fallback scopes) when looking up semantic token fallback scopes provided by extensions, so ideally there should be no additional work in the C/C++ Extension to address the issue.

I do see that the following section was recently added to the document here.

The provider of a TextMate grammar can describe the language-specific scopes. That helps with themes that contain language-specific theming rules.

{
  "contributes": {
    "semanticTokenScopes": [
      {
        "language": "typescript",
        "scopes": {
          "property.readonly": ["variable.other.constant.property.ts"]
        }
      }
    ]
  }
}

However, our extension is not a TextMate grammar provider...

As an experiment, could you try replacing the semanticTokenScopes section in our package.json with the following JSON, and let me know if that appears to address the issues you are seeing? You should be able to find the installed extension under the .vscode directory in your user directory. i.e. On Windows: C:\Users\<user>\.vscode\extensions\ms-vscode.cpptools-1.5.0-insiders3\package.json

JSON (Click to expand)

    "semanticTokenScopes": [
      {
        "language": "c",
        "scopes": {
          "namespace": [
            "entity.name.namespace.c"
          ],
          "type": [
            "entity.name.type.c"
          ],
          "type.defaultLibrary": [
            "support.type.c"
          ],
          "struct": [
            "storage.type.struct.c"
          ],
          "class": [
            "entity.name.type.class.c"
          ],
          "class.defaultLibrary": [
            "support.class.c"
          ],
          "interface": [
            "entity.name.type.interface.c"
          ],
          "enum": [
            "entity.name.type.enum.c"
          ],
          "function": [
            "entity.name.function.c"
          ],
          "function.defaultLibrary": [
            "support.function.c"
          ],
          "method": [
            "entity.name.function.member.c"
          ],
          "variable": [
            "variable.other.readwrite.c",
            "entity.name.variable.c"
          ],
          "variable.readonly": [
            "variable.other.constant.c"
          ],
          "variable.readonly.defaultLibrary": [
            "support.constant.c"
          ],
          "parameter": [
            "variable.parameter.c"
          ],
          "property": [
            "variable.other.property.c"
          ],
          "property.readonly": [
            "variable.other.constant.property.c"
          ],
          "enumMember": [
            "variable.other.enummember.c"
          ],
          "event": [
            "variable.other.event.c"
          ],
          "label": [
            "entity.name.label.c"
          ],
          "variable.global": [
            "variable.other.global.c"
          ],
          "variable.local": [
            "variable.other.local.c"
          ],
          "property.static": [
            "variable.other.property.static.c"
          ],
          "method.static": [
            "entity.name.function.member.static.c"
          ],
          "macro": [
            "entity.name.function.preprocessor.c",
            "entity.name.function.macro.c"
          ],
          "referenceType": [
            "entity.name.type.class.reference.c"
          ],
          "cliProperty": [
            "variable.other.property.cli.c"
          ],
          "genericType": [
            "entity.name.type.class.generic.c"
          ],
          "valueType": [
            "entity.name.type.class.value.c"
          ],
          "templateFunction": [
            "entity.name.function.templated.c"
          ],
          "templateType": [
            "entity.name.type.class.templated.c"
          ],
          "operatorOverload": [
            "entity.name.function.operator.c"
          ],
          "memberOperatorOverload": [
            "entity.name.function.operator.member.c"
          ],
          "newOperator": [
            "keyword.operator.new.c"
          ],
          "numberLiteral": [
            "entity.name.operator.custom-literal.number.c"
          ],
          "customLiteral": [
            "entity.name.operator.custom-literal.c"
          ],
          "stringLiteral": [
            "entity.name.operator.custom-literal.string.c"
          ]
        }
      },
      {
        "language": "cpp",
        "scopes": {
          "namespace": [
            "entity.name.namespace.cpp"
          ],
          "type": [
            "entity.name.type.cpp"
          ],
          "type.defaultLibrary": [
            "support.type.cpp"
          ],
          "struct": [
            "storage.type.struct.cpp"
          ],
          "class": [
            "entity.name.type.class.cpp"
          ],
          "class.defaultLibrary": [
            "support.class.cpp"
          ],
          "interface": [
            "entity.name.type.interface.cpp"
          ],
          "enum": [
            "entity.name.type.enum.cpp"
          ],
          "function": [
            "entity.name.function.cpp"
          ],
          "function.defaultLibrary": [
            "support.function.cpp"
          ],
          "method": [
            "entity.name.function.member.cpp"
          ],
          "variable": [
            "variable.other.readwrite.cpp",
            "entity.name.variable.cpp"
          ],
          "variable.readonly": [
            "variable.other.constant.cpp"
          ],
          "variable.readonly.defaultLibrary": [
            "support.constant.cpp"
          ],
          "parameter": [
            "variable.parameter.cpp"
          ],
          "property": [
            "variable.other.property.cpp"
          ],
          "property.readonly": [
            "variable.other.constant.property.cpp"
          ],
          "enumMember": [
            "variable.other.enummember.cpp"
          ],
          "event": [
            "variable.other.event.cpp"
          ],
          "label": [
            "entity.name.label.cpp"
          ],
          "variable.global": [
            "variable.other.global.cpp"
          ],
          "variable.local": [
            "variable.other.local.cpp"
          ],
          "property.static": [
            "variable.other.property.static.cpp"
          ],
          "method.static": [
            "entity.name.function.member.static.cpp"
          ],
          "macro": [
            "entity.name.function.preprocessor.cpp",
            "entity.name.function.macro.cpp"
          ],
          "referenceType": [
            "entity.name.type.class.reference.cpp"
          ],
          "cliProperty": [
            "variable.other.property.cli.cpp"
          ],
          "genericType": [
            "entity.name.type.class.generic.cpp"
          ],
          "valueType": [
            "entity.name.type.class.value.cpp"
          ],
          "templateFunction": [
            "entity.name.function.templated.cpp"
          ],
          "templateType": [
            "entity.name.type.class.templated.cpp"
          ],
          "operatorOverload": [
            "entity.name.function.operator.cpp"
          ],
          "memberOperatorOverload": [
            "entity.name.function.operator.member.cpp"
          ],
          "newOperator": [
            "keyword.operator.new.cpp"
          ],
          "numberLiteral": [
            "entity.name.operator.custom-literal.number.cpp"
          ],
          "customLiteral": [
            "entity.name.operator.custom-literal.cpp"
          ],
          "stringLiteral": [
            "entity.name.operator.custom-literal.string.cpp"
          ]
        }
      },
      {
        "language": "cpp-cuda",
        "scopes": {
          "namespace": [
            "entity.name.namespace.cpp-cuda"
          ],
          "type": [
            "entity.name.type.cpp-cuda"
          ],
          "type.defaultLibrary": [
            "support.type.cpp-cuda"
          ],
          "struct": [
            "storage.type.struct.cpp-cuda"
          ],
          "class": [
            "entity.name.type.class.cpp-cuda"
          ],
          "class.defaultLibrary": [
            "support.class.cpp-cuda"
          ],
          "interface": [
            "entity.name.type.interface.cpp-cuda"
          ],
          "enum": [
            "entity.name.type.enum.cpp-cuda"
          ],
          "function": [
            "entity.name.function.cpp-cuda"
          ],
          "function.defaultLibrary": [
            "support.function.cpp-cuda"
          ],
          "method": [
            "entity.name.function.member.cpp-cuda"
          ],
          "variable": [
            "variable.other.readwrite.cpp-cuda",
            "entity.name.variable.cpp-cuda"
          ],
          "variable.readonly": [
            "variable.other.constant.cpp-cuda"
          ],
          "variable.readonly.defaultLibrary": [
            "support.constant.cpp-cuda"
          ],
          "parameter": [
            "variable.parameter.cpp-cuda"
          ],
          "property": [
            "variable.other.property.cpp-cuda"
          ],
          "property.readonly": [
            "variable.other.constant.property.cpp-cuda"
          ],
          "enumMember": [
            "variable.other.enummember.cpp-cuda"
          ],
          "event": [
            "variable.other.event.cpp-cuda"
          ],
          "label": [
            "entity.name.label.cpp-cuda"
          ],
          "variable.global": [
            "variable.other.global.cpp-cuda"
          ],
          "variable.local": [
            "variable.other.local.cpp-cuda"
          ],
          "property.static": [
            "variable.other.property.static.cpp-cuda"
          ],
          "method.static": [
            "entity.name.function.member.static.cpp-cuda"
          ],
          "macro": [
            "entity.name.function.preprocessor.cpp-cuda",
            "entity.name.function.macro.cpp-cuda"
          ],
          "referenceType": [
            "entity.name.type.class.reference.cpp-cuda"
          ],
          "cliProperty": [
            "variable.other.property.cli.cpp-cuda"
          ],
          "genericType": [
            "entity.name.type.class.generic.cpp-cuda"
          ],
          "valueType": [
            "entity.name.type.class.value.cpp-cuda"
          ],
          "templateFunction": [
            "entity.name.function.templated.cpp-cuda"
          ],
          "templateType": [
            "entity.name.type.class.templated.cpp-cuda"
          ],
          "operatorOverload": [
            "entity.name.function.operator.cpp-cuda"
          ],
          "memberOperatorOverload": [
            "entity.name.function.operator.member.cpp-cuda"
          ],
          "newOperator": [
            "keyword.operator.new.cpp-cuda"
          ],
          "numberLiteral": [
            "entity.name.operator.custom-literal.number.cpp-cuda"
          ],
          "customLiteral": [
            "entity.name.operator.custom-literal.cpp-cuda"
          ],
          "stringLiteral": [
            "entity.name.operator.custom-literal.string.cpp-cuda"
          ]
        }
      }
    ]

If that content addresses the issue, we could potentially release it as a work-around. However, I believe the root issue here is a VS Code issue, and it should be responsible for appending language-IDs when looking up semantic token fallback scopes.

[REYNEP]

Oh, @Colengms, I am really sorry for Late reply, But, The changed semanticTokenScopes that you provided... seems to do the work!!!!

Thanks for making it work, sir. ☺️

[Colengms]

Hi @REYNEP . I opened the following issue against VS Code: microsoft/vscode#128565

Were you only seeing issues with semantic token fallback scopes, or were you seeing a similar issue in other scenarios?

[REYNEP]

Only the ones that C/C++ extensions used. I guess those are fallback scopes
Didn't encounter similar issue if Semantic Token scopes were not present

[Colengms]

This should be addressed in 1.6.0-insiders.