Support specifying language for semantic token types and modifiers

[0dinD]

Issue Type: Feature Request

Currently, only the semanticTokenScopes contribution point supports language-specific contributions for semantic tokens, via the "language": "..." property. But the semanticTokenTypes and semanticTokenModifiers contribution points do not support specifying the language, which means that semantic token types and modifiers are declared globally, for all languages. This causes a problem where the token contribution of one extension might override the token contribution of the same ID, from another extension (or even from the default tokens).

Implementing this feature would yield the following benefit:

Extensions providing semantic tokens could declare supertypes for new and existing token types, without breaking semantic highlighting for other languages that want to declare a different supertype for the same token type. For example, Java methods, which get the token type function in the redhat.java extension, should have the token supertype of member since that holds true for all methods in Java. But in other languages, such as TypeScript, functions are not always members. If the redhat.java extension were to declare member the supertype of function, this would cause the same behavior in TypeScript, which is undesirable. Instead, the redhat.java extension should be able to declare that functions are members, but only for tokens provided to Java files. Another "collision" that could happen in the future is that annotations are types in Java, but they are functions in TypeScript.

VS Code version: Code 1.47.2 (17299e4, 2020-07-15T18:22:15.161Z)
OS version: Linux x64 4.19.0-9-amd64

[aeschli]

I was hoping that custom token type or modifier are not the norm, but instead that the standard types and modifiers will be used as much as possible. That will allow themes and users to write theming rules that work across languages and result in consistent theming experience.
It's important to remember that theming is the main purpose, it's not about representing all concepts of a language in the languages own terms.

If a custom token type or modifier is very language specific, it makes sense to prefix it that way: javaMember (extends member). But I don't think javaMember is a good example of a very 'java' specific concept. member is already defined in the standard types vocabulary and I think should be used for Java methods. There's also function, and there's no inheritance defined, on purpose, for the reasons you named.

We have #97063 to discuss new standard tokens. annotation is a good suggestion. I see it as concept on it's own, no need to define it a subtype of type or function or whatever.

[0dinD]

I was hoping that custom token type or modifier are not the norm, but instead that the standard types and modifiers will be used as much as possible. That will allow themes and users to write theming rules that work across languages and result in consistent theming experience.
It's important to remember that theming is the main purpose, it's not about representing all concepts of a language in the languages own terms.

I agree with this, and my feature request is actually aimed at minimizing the need for theme authors to use language-specific token types, although it might not have been clear in my previous comment. Let me explain:

We seem to agree that providing only language-specific token types will be a hindrance for theme authors, and just a bad idea in general. But the problem that I see with generalizing all the token types is that it takes away capabilities from theme authors who do know about the more specific token types, and who want to use them. For example, let's say that member was used instead of the more specific function, property, enumMember and annotationMember token types in Java. Now, you cannot differentiate between a Java method, field, enum member or annotation member.

But as you say, with the more specific token types, now there is no way for theme authors to style member tokens, even though all the above are examples of members. That's why I wanted the ability to declare all these token types to also be of token type member, but only in Java. It makes both sides happy, as it becomes optional to use and know about language-specific token types.

After thinking some more about it, I do see another way of achieving the same goal. What the redhat.java extension could do is provide completely custom token types, prefixed by "java" like you said, and then declare supertypes on those tokens that relate them back to the standard token types. This way, "experts" can use javaMethod, javaField, javaAnnotationMember etc. and others can rely on these tokens being supertyped to member, property etc.

Do you think this is a better solution than my previous suggestion about being able to specify the language? Or do you think there is a better way to achieve the same goal? I am open to suggestions, but I really want to make a solution that works for both "experts" and those who want to rely on the standard token types.

Should I make a comment to suggest annotation in the discussion about new standard token types, or will you do that?

[0dinD]

Something I just realised about the second solution I had, is that I would really need to be able to declare multiple supertypes for the custom token types, in order for it to work like I wanted. Let me make a hypothetical list of "java" token types (some of which have not been implemented yet, but might come in the future):

• javaPackage -> namespace
• javaClass -> class, type
• javaInterface -> interface, type
• javaEnum -> enum, type
• javaAnnotation -> (annotation if it becomes standard), type
• javaTypeParameter -> typeParameter, type
• javaTypeArgument -> type
• javaMethod -> function, member
• javaConstructor -> javaMethod
• javaField -> property, member
• javaAnnotationMember -> javaMethod
• javaEnumMember -> javaField
• javaVariable -> variable
• javaParameter -> parameter, variable

Read the arrow as "extends". Do you feel like multiple supertypes with these custom tokens would be better than language-specific supertypes?

A third solution that might work is for the redhat.java extension to declare a setting, which enables or disables the more detailed token types, replacing them with the generalized ones. The only problem is, now theme authors using the detailed token types will have to tell users to enable the setting in order to make the theme look correct. What do you think?

[aeschli]

You can add Java specific types if that solves a user request for you, but note that users/theme can already define language specific theming rules: "variable:java": { "bold:": true }

member is only intended for member functions. Maybe a clearer name would have been memberFunction.

The point of the inheritance is that you can inherit styling rules of other types. I don't know what it would mean if multiple inheritance would be allowed. So I'd rather not go there and avoid more complexity.

[0dinD]

Yes, I knew about the language specifier in the theming rules. The examples above were just a naming scheme to indicate that those tokens had a different meaning/hierarchy than the standard token types.

member is only intended for member functions. Maybe a clearer name would have been memberFunction.

Oh, that clears many things up for me! I was really confused about the meaning of the member token and the way you wanted to apply it. In that case, I guess it would be fine to just use the standard token types in most cases. I do agree that it will create a better theming experience if the standard tokens can be used in every language.

I understand if you don't want to add more complexity to the system, and now that you cleared things up I no longer feel like we need this specific feature. But if you are not planning to change the name of member, I think you should at least clearly document the intended meaning of it, to avoid more confusion in the future. Theme authors probably find it equally confusing, memberFunction is a much better name for what purpose you described. Thank you for clarifying! If you don't have anything to add, feel free to close this issue.

[aeschli]

Thanks for your efforts. I realize now that member is confusing. I want to document all types in microsoft/vscode-docs#3522