How can I use tree-sitter test with an external scanner not written in C/C++?

[CAD97]

I'm writing a parser with a scanner written in Rust. From the Rust bindings, everything is working to link.

tree-sitter test, however, fails to link, because it's not linking in the scanner code. I've not told the node bindings anything about the Rust implemented scanner, so this isn't too surprising... but how do I do so?

The binding.gyp has a clear place to add new C/C++ files for a C/C++ scanner, but is it possible to get Rust linked into the node bindings?

I don't really need/intend this parser to be usable from node (but it'd be nice for sure), but I do want to be able to use tree-sitter test.

[ahelwer]

Not answering your question, but do note every project that currently consumes tree-sitter uses a C/C++ toolchain to build the parser so writing your scanner in rust bars you from integrating with them. Things like neovim and such.

[ruifengx]

There is a link_settings section available in binding.gyp, and we can add libraries to link here. (See for example here for the syntax.) I also found this repository which specifically deals with linking to Rust in a node module as a dynamic library.

I think we can do the following:

• Create a Rust scanner crate and build as cdylib;
• Specify the dependency in binding.gyp as shown here;

Haven't tested it yet, but I presume it should work (though a bit ugly IMO). The Rust scanner should be built as cdylib rather than a staticlib because I believe we will need to manually link libstd ourselves otherwise. However this is only the case when linking for node; when using the parser from Rust we should be able to link as rlib as usual.

---

EDIT: it seems libstd is included in staticlibs, so it might also just work with staticlib (and if it does I think this is preferable).

---

Actually this does not work. I checked the implementation of the CLI just now, and it just hard-coded the path scanner.c/scanner.cc in the loader. The gyp file is only used for building the binding for node. See the following line for details:

tree-sitter/cli/loader/src/lib.rs

Line 314 in b268e41

let mut scanner_path = src_path.join("scanner.c");

We will need a PR targetting this part to support non C/C++ scanners in the test command.

[maxbrunsfeld]

I'd love to make it possible for tree-sitter test and tree-sitter parse to compile and link external scanners written in Rust. If someone wants to open a PR, I'd be open to it.

[ahelwer]

After fixing a ridiculous bug in my C++ external scanner I am now dying to RIIR lol