Tree-sitter doesn't work with one whitespace character. [Bug?]

[urbit-pilled]

Problem

I am attempting to create a simple grammar that takes as input and array of n numbers, like so [1 2], where the format is "[" $.number $.space $.number "]". It should take as input only 1 space inbetween the number.

Expected output

[1 2] should generate (array (number) (space) (number))
[1 2] should generate (array (number) (space) (space) (space) (number))

Actual output

[1 2] generates (array (number) (space) (number))
[1 2] generates (array (number) (space) (number))

Minimal grammar file

module.exports = grammar({
    name: "SimpleArray",
    rules: {
        source_file: $ => seq($._expression, repeat(seq($.space, $._expression))),
        extras: $ => '',

        _expression: $ => choice(
            $.number,
            $.array
        ),

        array: $=> choice(
            seq("[", $._expression, repeat(seq($.space, $._expression)), "]")
        ),

        number: $ => /[0-9]+/,
        space: $ => ' ',
    }
});

[sogaiu]

If handling whitespace explicitly is desired, it might make sense to check out extras or writing an external scanner.

There is some info about this summarized recently starting here, but it might make sense to check out the rest of the discussion, specifically comments from @ahlinc (e.g. #2159 (reply in thread)).

[urbit-pilled]

I recall that the default extras is to ignore whitespace characters, which is why I set extras to an empty string ''.
If I'm reading the threads correctly, is it correct to assume that the only way to handle a single whitespace character is to write an external scanner? And token matching for " " doesn't work properly at all?

[sogaiu]

Ah sorry, I missed that you had set extras.

Though I'm not sure how it functions when set to a function that returns an empty string.

I set it to a function which returns [] and handle whitespace explicitly so I don't think an external scanner is required. For the case I linked to, the grammar is relatively straight-forward so an external scanner was not implemented. Though it's not handling exactly single spaces at a time.

IIUC what ahlinc mentioned, depending on the rest of one's grammar it might be the easier route to go with an external scanner.

[sogaiu]

Oh, may be it's that extras should be outside of rules?

[urbit-pilled]

Oh, thanks @sogaiu. That was a good catch, extras is meant to be set outside rules.

[urbit-pilled]

@sogaiu solved this problem. #2165 (reply in thread)