[SR-9616] SwiftSyntax is not following trivia rule

[swift-ci]

Previous ID	SR-9616
Radar	None
Original Reporter	inamiy (JIRA User)
Type	Bug

Attachment: Download

Environment

• Apple Swift version 4.2.1 (swiftlang-1000.11.42 clang-1000.11.45.1)

• SwiftSyntax 0.40200.0

Additional Detail from JIRA

Votes	0
Component/s	Compiler, SwiftSyntax
Labels	Bug, Parser, Syntax
Assignee	None
Priority	Medium

md5: b19e5c1a45f2ad0872b99a66f9211c27

relates to:

• SR-2373 Trailing doc comment should not be attached to the next declaration

Issue Description:

According to the libSyntax documentation:

1. A token owns all of its trailing trivia up to, but not including, the next newline character.
2. Looking backward in the text, a token owns all of the leading trivia up to and including the first contiguous sequence of newlines characters.

The following code should have leading / trailing trivias as follows:

let x = 1 // This should be "1"'s trailing trivia
/// This should be `var`'s leading trivia
var y = 2

But actually, both comments are bundled as `var`'s leading trivia.

You can test the code at https://swift-ast-explorer.kishikawakatsumi.com/ (screenshot attached).

By the way, in TypeScript, it seems to work as is:
https://github.com/basarat/typescript-book/blob/master/docs/compiler/ast-trivia.md

[rintaro]

I agree this is a bug.

https://github.com/apple/swift/blob/75033ed05a8f5ecf79e77b41dabd2c3863cbc9e5/lib/Parse/Lexer.cpp#L2560
I intentionally did this because, currently, we want such comments to be attached to the next token (i.e. possible start of a declaration on the next line).

https://bugs.swift.org/browse/SR-2373

Lexing trailing-comment as trailingTrivia makes parsing difficult as long as we keep current "trailing doc-comment" behavior. @nkcsgexi @akyrtzi What do you think?

[nkcsgexi]

This example inamiy (JIRA User) gave here is an obvious bug since the comments are in different forms ("//" and "///"), so it's reasonable to assume they're
commenting different pieces.

If they are both "///", i have no strong opinion on whether we split them as trailing and leading trivia or combine them as leading trivia, because the lexer cannot
be always right in either way.

[swift-ci]

Comment by Yasuhiro Inami (JIRA)

@nkcsgexi

If they are both "///", i have no strong opinion on whether we split them as trailing and leading trivia or combine them as leading trivia, because the lexer cannot
be always right in either way.

I think trailing and leading trivias should be separated in the above case too, and it's probably possible to distinguish them by checking if token exists in the same line or not.

If this can be done, it's exciting to see SR-2373 trailing-doc support can be implemented as well 🙂