feat: rewrite the scanner in C

[amaanq]

No description provided.

[vkleen]

Do you have a particular rationale for rewriting the scanner in C instead of C++?

[amaanq]

Portability and ease of compilation - it better supports projects using a pure C toolchain and the logic is simple enough to not really need the C++ containers

[yannham]

Hi, and thanks for the contribution!

IMHO inlining the implementation of vector using a custom struct and C macros isn't a reasonable tradeoff. If @vkleen or anyone from the Topiary team reports that there is strong evidence that this fixes their issues for compiling Topiary as a static binary for example, or that it might solve someone's actual and very concrete issue, then it might be considered. Until then I would personally not endorse this change based on a "this might probably help with", considering it doesn't have a null cost and that it doesn't fix any known bug directly (put differently, "if it ain't broke, don't fix it")

[amaanq]

Sure, those are valid concerns - but I've been rewriting many C++ scanners across various tree-sitter repos - for the most part, it doesn't solve a particular bug and it really is just about making compilation easier and improving portability. Though in some cases, there were bugs with leaks or buffer overflows in the original scanner that I had fixed anyways (not this one)

[amaanq]

I'll also tack on that building this for wasm is much easier than with C++ due to the need to manually specify specific std functions to export (in tree-sitter core there is a file that manually picks common functions to export)

Anyways I updated it to add static to any helper functions

[clason]

I'll also add that upstream has officially deprecated C++ scanners, and such parsers will be removed in nvim-treesitter 1.0.

[yannham]

Thanks for the information, @clason. I think this is a good argument in favor of moving the scanner to C, despite the loss of some nice abstractions.

[clason]

Upstream is adding some nice abstractions to the C headers, though, so it might not be too bad: tree-sitter/tree-sitter#3063

[yannham]

Thanks again for undertaking that. We'll probably need to rebase and regenerate the parser, as there's a conflict currently.

[amaanq]

Thanks again for undertaking that. We'll probably need to rebase and regenerate the parser, as there's a conflict currently.

C++ is a pain to get working cross-platform and in wasm especially, and there's really no need for most of its abstractions. The only used abstractions across grammars with a scanner is vectors/strings, which the near-future inclusion of an array header will provide a decent replacement for.

[amaanq]

I've rebased now @yannham