Tokenizer splitting ..< when a comment follows

[angelolloqui]

This code:

1..<5

Results in this tokens:

po formatter.tokens
▿ 3 elements
  ▿ 0 : Token
    - number : "1"
  ▿ 1 : Token
    - symbol : "..<"
  ▿ 2 : Token
    - number : "5"

Which looks OK.

However, when adding a comment after the expression, then the tokenizer replaces .symbol("..<") with 2 tokens, a .symbol("..") and a .startOfScope("<"). Example:

1..<5
//comment

Results in:

po formatter.tokens
▿ 7 elements
  ▿ 0 : Token
    - number : "1"
  ▿ 1 : Token
    - symbol : ".."
  ▿ 2 : Token
    - startOfScope : "<"
  ▿ 3 : Token
    - number : "5"
  ▿ 4 : Token
    - linebreak : "\n"
  ▿ 5 : Token
    - startOfScope : "//"
  ▿ 6 : Token
    - commentBody : "comment"

The start of scope is a very delicate token because it makes other searches break, as everything after the < is in a different scope now.

[nicklockwood]

Nice catch!

By all means have a go at fixing it, or I'll tackle it myself this evening. Either way, a PR with a unit test to reproduce the issue would be much appreciated!

[nicklockwood]

Fixed in 0.17.1