tree-sitter get wrong point when rules contains '\s'

[starwing]

With this test grammar:

module.exports = grammar({
  name: 'test',

  rules: {
    // TODO: add the actual grammar rules
    source_file: $ => $.program,

    program: ($) =>
      prec(16,
        repeat1(choice(
          $._statement,
          $._document,
        ))
      ),

    _statement: ($) =>
      prec.right(15,
        seq($.identifier)
      ),
   
      identifier: ($) => /[a-zA-Z_][a-zA-Z0-9_]*/,
      _document: ($) => /foo/,  // <-- notice this
  }
});

and the test input:

a
b
c

tree-sitter produces this result (correct):

$ tree-sitter parse test
(source_file [0, 0] - [2, 1]
  (program [0, 0] - [2, 1]
    (identifier [0, 0] - [0, 1])
    (identifier [1, 0] - [1, 1])
    (identifier [2, 0] - [2, 1])))

but if I changed document to this:

      _document: ($) => /\sfoo/, // <-- notice this

this result changed to this:

$ tree-sitter parse test
(source_file [0, 0] - [2, 1]
  (program [0, 0] - [2, 1]
    (identifier [0, 0] - [0, 1])
    (identifier [0, 1] - [1, 1])
    (identifier [1, 1] - [2, 1])))

even there is not any "foo" in test input. the "newline" before identifiers is included into identifier.

[amaanq]

newlines are an extra by default (always skipped if possible wherever), if you don't want that then specify your extras property as an empty array

I misunderstood your question, this is interesting

I believe it has to do w/ the regex starting to process \s, then failing and trying to recover. See the output of the good and bad tree:

Good:

Bad:

Regexes are always terminal, and I'd like to say the usage of \s here causes the parser to try and not treat ws in possible identifier/document scenarios as an extra

[starwing]

@amaanq Thanks for the explain, so it is a bug or just a feature?

[amaanq]

Neither, feature isn't the right word imo, more like intended behavior