fix: editing subtree that depends-on-column: ...

[rooney]

... invalidation should be based on the edit's end-column.

Fix #3217

[maxbrunsfeld]

Hi, I don't know what behavior this new code would implement, but the existing behavior is what we want, I believe.

The existing behavior is that if the external scanner's get_column method is called when scanning a token, then any edit that occurs before that token on a line (thus shifting the token's start column) will invalidate the token. This is, of course, in addition to the normal invalidation behavior, where tokens are invalidated if an edit intersects the range of text that was read when scanning them.

[maxbrunsfeld]

I'm going to close this out because I believe the existing test is correct, and your change to the test would break the desired behavior. If I'm missing some more subtle reason why this invalidation logic should be changed, then feel free to explain further, and we can reopen.

[rooney]

Hi @maxbrunsfeld

Alright, I have reworked my PR so it's much simpler now -- only adding two checks and nothing else (no test change, and all the tests still pass).

master...rooney:tree-sitter:col-dependent-fix
(I assume it's pretty clear since it's only a two-liner, but please let me know if I should explain)

The change is needed because there are some cases where a token-that-depends-on-column should be invalidated but isn't. Let me demo it using a simplest-as-possible example:

// grammar.js

  rules: {
    source_file: ($) => seq(/[ \t\r\n]+/, choice($.odd, $.even), $.x),
    x: () => "x",
  },
  externals: ($) => [$.odd, $.even],

Here, our grammar will: 1. consume leading whitespace, 2. consult the external scanner to see if the column is at odd/even position, and 3. match "x".

(the scanner is stateless):

// scanner.c

enum TokenType { ODD, EVEN };

bool tree_sitter_coldemo_external_scanner_scan(void *payload, TSLexer *lexer,
                                               const bool *valid_symbols) {
  lexer->result_symbol = lexer->get_column(lexer) % 2 ? ODD : EVEN;
  return true;
}

Now, here's how the playground goes on master:

See how it stays "odd" even though the positions are correctly updating.

And here's how the playground goes after the two-line patch:

See how it correctly updates the odd/even now.

tree-sitter-coldemo.zip (contains only grammar.js and src/scanner.c)

To be extra sure, I also have checked this new patch against the complete tree-sitter-haskell -- all tests there still pass running this patch.

[maxbrunsfeld]

Thanks for the response, and nice work. As part of your branch, can you add your new example scanner as a test to the Tree-sitter test suite?

[maxbrunsfeld]

Reopening.

[rooney]

Reopening.

Thanks @maxbrunsfeld

• OK test added, made sure it doesn't pass without the patch.

Also, added an optimization for when the edits don't actually move the columns. (The test for this optimization checks that there's fewer bytes being read by the recorder).
master...rooney:tree-sitter:col-dependent-fix

Also have re-run tree-sitter-haskell test with the optimization, all good there.

[amaanq]

I don't think it can't be reopened, can you open another PR @rooney?

[maxbrunsfeld]

Oh sorry, thought that I reopened it. The GitHub UI mislead me a bit there.

[rooney]

sure, np, created another PR #3257