[SR-10024] Add new TokenKinds for contextual keyword in libSyntax/SwiftSyntax

[swift-ci]

Previous ID	SR-10024
Radar	None
Original Reporter	kitasuke (JIRA User)
Type	Improvement

Attachment: Download

Additional Detail from JIRA

Votes	0
Component/s	Compiler, SwiftSyntax
Labels	Improvement
Assignee	None
Priority	Medium

md5: f6a7999be8142907c0d76093517b06c8

Issue Description:

All contextual keyword's token kind is `identifier` for now, but I would like to have them as keyword kind instead.

For example, I was expecting to have `weakKeyword` for below variable declaration, but I got `identifier("weak")` in SwiftSyntax. In this context, it can tell keyword, not identifier.

Same thing happens to other contextual keywords as well.

@IBOutlet private weak var button: UIButton!

Sample code:

// sample.swift
class View {
 @IBOutlet private weak var button: UIButton!
}

 $swiftc -frontend -emit-syntax sample.swift  

Expected:

"tokenKind":{"kind":"kw_weak"} 

Actual:

"tokenKind":{"kind":"identifier","text":"weak"}

Environment:

Xcode 10.2 beta 3 (10P99q)

Apple Swift version 5.0 (swiftlang-1001.0.63.8 clang-1001.0.43)
Target: x86_64-apple-darwin18.2.0

[rintaro]

cc: @nkcsgexi @akyrtzi

[akyrtzi]

I think this is the kind of contextual "classification" we have the SyntaxClassifier for.

For example for "let x: Int" "Int" is an identifier token, but the classifier will mark it as 'type-identifier'. In the case of "weak" in your example it will mark it as 'builtin-attribute' classification kind.

[rintaro]

We already have ContextualKeyword token kind. We probably should use them for modifiers at least. WDYT @akyrtzi?

[akyrtzi]

Could you clarify how we use it currently ?

[rintaro]

In parser, we set the token kind to tok::contextual_keyword for several places such as accessor introducer (get, set, willSet, etc.), #selector, or attribute names for precedencegroup (higherThan etc.). But not for decl modifiers.

[rintaro]

@harlanhaskins Do you remember why ContextualKeyword was introduced? It's not entirely clear from swiftlang/swift#13321

[harlanhaskins]

That patch was really introducing TokenKinds to cover the new ones in swiftlang/swift#11809

I think SyntaxClassifier serves the need that contextual_keyword served. @nkcsgexi should we get rid of contextual_keyword?

[akyrtzi]

I've added a convenience API TokenSyntax.tokenClassification to get the classification for a specific token (#109 I think we should get rid of contextual_keyword.

[ahoppen]

rdar://95638944

[ahoppen]

Looks like weak in the example is now correctly classified as a contextual keyword.

~> cat /tmp/a.swift
class View {
 @IBOutlet private weak var button: UIButton!
}
~> ~/sbin/swift-frontend -frontend -emit-syntax /tmp/a.swift
…
{
    "kind": "contextual_keyword",
    "text": "weak"
},
...