Preserve raw tokens for strings and regexps

[dennwc]

Currently, JS driver uses unescaped string and regexp literals from the native AST as @token. This leads to confusion for ML, since they expect string token to cover the whole node in the source code (according to the positional information).

Although in general, this will require adding a separate "token stream" to the UAST, those two issues can be solved in the current version.

See comments in related issues for details (link is in the commit message).

[bzz]

👍 If I understood the change, it boils down to keeping whole literals in @token:

Code: string literal attribute="If parent is JSX keep double quote"
Before

• Node @token If parent is JSX keep double quote
  After
• Node @token "If parent is JSX keep double quote"

and same for
Code: regexp literal /a/g
Before

• Node @token a
  After
• Node @token /a/g

@dennwc could you please help me understand, how/if we want it to be consistent with behavior of other drivers?

On

See comments in related issues for details.

not sure which issue is that.

On driver/normalizer/strconv.go- would it make sense to have simple tests for those functions?

[dennwc]

@bzz Other drivers already include quotes in @token for Native and Annotated mode. This is only an issue with the JS driver that uses wrong fields from native AST.

As for strconv.go, I don't think it requires extensive test coverage since it's copied from the Go stdlib, which already ensures that the code is correct. But we can add a few small test cases to detect regressions.

[bzz]

Thank you for kind explanations!

On

See comments in related issues for details.

not sure which issue is that.

Have missed those in commit messages - #22 and #32

[bzz]

👏