Proper way to get full syntax highlighting when building grammar into a library for consumption?

[sortofsleepy]

👋

Apologies if this isn't the right spot to ask, but figured this might be the right place to start at least. It's likely a skill / knowledge issue on my part but the documentation on how this particular aspect of Treesitter works is a bit lacking and I was hoping someone could help clear things up.

tldr' - I can build a library for use in Emacs just fine; but syntax highlighting isn't complete and only some parts are highlighted.

Setting the scene

• I'm on Windows 11
• Trying to compile grammars for Emacs 29.2
• I have node and the Treesitter cli is installed via cargo
• When I reference "pre-built" libraries, I'm talking specifically about this one

My current understanding of the process is as follows

• Install cli (not sure if there are differences between NPM and Cargo versions)
• clone grammar repository
• Inside the grammar repo, run npm install, then tree-sitter generate
• Compile parser.c (and scanner.c if present) into appropriate library format with some kind of c compiler depending on system or run tree-sitter generate -b in the previous step instead.

After fiddling around with things I'm about 98% there - While using the cli, testing, highlighting, etc helpers all work as expected.

However the missing 2% is the syntax highlighting inside of Emacs. I can successfully build a library for use inside of Emacs and while there is "some" highlighting, there is not as much compared to using one of the pre-built libraries out there or the cli highlighter.

Obviously I'm missing a step and I'm not sure where to go from here.

I should note that running the cli's generate -b command appears to result in the same library behavior compared to when I compile things myself.

Any help is appreciated! Thank you!🍻

For comparison

First is using pre-built, second is the version I built

Possibly related?

• It's not clear how the config file(generated via tree-sitter init-config) is utilized. It does look like that's potentially part of the issue(?)
• From what I understand, the queries folder of a grammar stores the important syntax highlighting bits but it's not clear how to compile the scm files into the library.
• On Windows, when I run npm install I see Node generates a VS studio project - When I look at what it automatically builds, I get a bunch of build artifacts including a .node file which appears to be what the project is supposed to output to(this also happens when I open/build the project myself). Trying to use this file however results in Emacs not being able to find the grammar library, even if I rename it to a .dll (which is what the project is supposed to generate anyways)

Last Notes

I realize there are alternatives to building grammars myself, however

• With the built-in function within Emacs 29, for some reason, while it manages to create the appropriate temporary directories, it somehow isn't able to clone a grammar repo to the directory and fails midway. Not sure if it's a Windows / Emacs issue or not and not sure how to debug. This is more a combination of user error + unexpected repo setup as it turns out. I initially ran into this error with the Typescript grammar which splits things into typescript + tsx and it didn't occur to me that that I wasn't accounting for that. I could have swore I tried this with a single language grammar as well but maybe not.

• I know pre-built libraries exist for Emacs - I just wanted to better understand the process myself from a more general perspective rather than relying on specific tooling(which is what the existing script that's available is doing). I will likely just end up running the script that's provided and see what happens but my Elisp knowledge isn't great yet and I'm not sure I'd fully understand what's happening without a bunch more debugging)

[amaanq]

Highlighting typically happens from using queries to match on certain sexp patterns and apply a capture/highlight to them. I would ask this in emacs wherever relevant as to how this is done, but you can test out highlighting with the highlight subcommand in a grammar dir.

[sortofsleepy]

Ah, ok - just to make sure I understand, you're saying that highlighting is usually left up to the program/plugin that utilizes Treesitter?

And yes, if I use the CLI, highlighting works fine

[amaanq]

just to make sure I understand, you're saying that highlighting is usually left up to the program/plugin that utilizes Treesitter?

yes

[sogaiu]

Also, for Emacs >= 29.1 with tree-sitter library support compiled in [1], typically highlighting is "programmatically" provided as part of a tree-sitter specific mode that *-ts-mode.el files [2] implement. The .scm files often don't come into it IIRC.

If you look for treesit-font-lock-rules, for example like this, may be you can get a sense of things.

FWIW, I don't see much discussion at this repository about using tree-sitter with Emacs. I've seen discussion at the gnu emacs devel mailing list though. There is also a discord server (mentioned at the top of here), but I don't know how many Emacs-using types hang out there ATM.

---

[1] This stuff is used by the Rust bindings effort for tree-sitter support for Emacs. AFAIU, it's not what Emacs >= 29.1 + tree-sitter stuff uses.

Also, the queries directory you might see in a specific grammar repository may often not be relevant (especially for Emacs >= 29.1 with tree-sitter support compiled in...need a shorter way to say this (^^; ) because each program that uses a query (e.g. an editor) tends to have its own versions of predicates (e.g. nvim-treesitter has some that IIUC others do not have).

[2] There may be exceptions -- this is just the way it had been for a while.

[sortofsleepy]

typically highlighting is "programmatically" provided as part of a tree-sitter specific mode that *-ts-mode.el files [2] implement. The .scm files often don't come into it IIRC

That did seem to be how the Nvim folks are doing it but I wasn't sure, especially given that someone's apparently managed to get better syntax highlighting bundled directly in somehow on the Emacs side.

If you look for treesit-font-lock-rules, for example like this, may be you can get a sense of things.
FWIW, I don't see much discussion at this repository about using tree-sitter with Emacs. I've seen discussion at the gnu emacs devel mailing list though. There is also a discord server (mentioned at the top of here), but I don't know how many Emacs-using types hang out there ATM.

Appreciate the links, will take a look

[1] This stuff is used by the Rust bindings effort for tree-sitter support for Emacs. AFAIU, it's not what Emacs >= 29.1 + tree-sitter stuff uses.

That looks like it's from the same repo I pulled the pre-built dlls from but interesting point. I managed to get the native Emacs functionality working and it seems like it builds something very similar to what I built (as it gives me the exact same poor syntax highlighting)
so that repo is probably something I might have to look more into (though I can't get their build script to work on my end ugh ha)

Also, the queries directory you might see in a specific grammar repository may often not be relevant (especially for Emacs >= 29.1 with tree-sitter support compiled in...need a shorter way to say this (^^; ) because each program that uses a query (e.g. an editor) tends to have its own versions of predicates (e.g. nvim-treesitter has some that IIUC others do not have).

will keep this in mind.

[sogaiu]

That did seem to be how the Nvim folks are doing it but I wasn't sure, especially given that someone's apparently managed to get better syntax highlighting bundled directly in somehow on the Emacs side.

AFAIU, neovim folks DO use .scm files -- the ones they use are specifically tailored for their use. I think a small number currently live with neovim itself and many many more live at nvim-treesitter's repository.

The "newer" Emacs stuff (>= 29.1 compiled with the C tree-sitter library) takes an approach that's quite different. Currently [1] the *-ts-mode.el files listed here contain bits that are used to dynamically construct what's comparable to what lives in a .scm file. This can give users some say in how they want things handled via configuration (e.g. don't highlight certain constructs).

That looks like it's from the same repo I pulled the pre-built dlls from but interesting point. I managed to get the native Emacs functionality working and it seems like it builds something very similar to what I built (as it gives me the exact same poor syntax highlighting)
so that repo is probably something I might have to look more into (though I can't get their build script to work on my end ugh ha)

I think some people use the repository you mentioned for grammar shared objects even with Emacs >= 29.1 built with tree-sitter library support. I've not tried this though.

I use this other one, but it's not particularly oriented toward non-unixy use.

Also, some major modes will do the fetching, compilation, and installation for you, e.g. like this in clojure-ts-mode. So depending on your situation, you might not need to do much.

I don't know how well that works on Windows though.

There's more detail on some of this here, though it may be that switching branches will yield something more relevant to your particular situation.

Good luck :)

---

[1] Exceptions may exist, not sure whether things will change in the future, etc.

[sortofsleepy]

AFAIU, neovim folks DO use .scm files -- the ones they use are specifically tailored for their use...

Sorry, sleep deprived brain kicking in then, could have phrased that better .. right right I was more referencing that it looked like they were somehow handling highlighting themselves programmatically like you and Amaan were saying, not that they weren't using scm files.

Appreciate the other links!

I need to test more but I think I actually just managed to figure it out just now. I somehow missed this detail initially but after reading the readme again, tree-sitter-langs appears to have altered .scm files which seems to influence the amount of syntax highlighting; I'm guessing by targeting different font-lock rules.

Tried with their highlights file and it seems to give me much better results; though they appear to also take different commits of various languages and not target the default branch as well which might have something to do with things too.

Ok - so I think the reason tree-sitter-langs has different results is more due to the different commits of each grammar that they're taking; if I build with those specific commits I get much better syntax highlighting now. So I'm guessing something must have changed in the parser between those commits and the current main commit for the particular languages I was testing.

Anyways I think I have enough to go on from here Mystery solved!.

Again appreciate the help and extra info! 🍻

[sortofsleepy]

Closing for now then since it sounds like this isn't really a Treesitter specific issue.