get_column acting weird before unicode symbols.

[banacorn]

... a ε b
    ^

Suppose that lexer->get_column(lexer) returns with some value n after advancing the character "a".

After skipping one whitespace with lexer->advance(lexer, true) and stops before some unicode symbol "ε", lexer->get_column(lexer) should return n + 1.

... a ε b
     ^

However, what I got was n - 1.

[maxbrunsfeld]

Wow that's not good. If you get a chance, could you provide a minimal example grammar and scanner that reproduces the problem?

[banacorn]

Here's an example repo (on brnach tree-sitter#312)

To run the example:

git clone git@github.com:tree-sitter/tree-sitter-agda.git
cd tree-sitter-agda
git checkout tree-sitter#312
npm install
npx tree-sitter generate
npx tree-sitter test

And you should see some numbers printed from the scanner

column before []: 3

column before []: 3

column before [B]: 4

column before [B]: 4

column before []: 3

column before []: 3

column before [D]: 4

column before [D]: 4

column before [E]: 4

column before [E]: 4

Here's the testing corpus:

    A
    ε
    B
    εC
    Dε
    E

I'm not sure how to make this example better, but I think you can see the problem.

[maxbrunsfeld]

Closing this one out in favor of #314. Unfortunately get_column is still a work in progress. We added it for use by the haskell grammar, but never prioritized completing the implementation. Let's leave #314 open to track this problem.