Automatically install tree-sitter queries (if they exist) on grammar build

[aral]

Use case

I create, fetch, and build a new tree-sitter grammar. I launch Helix Editor to see my new tree sitter grammar in action.

What should happen

I should see it working.

What actually happens

Nothing.

How I troubleshoot

I enter hx --health my_lang and I see that highlighting and indentation are not being applied for it.

I look inside my tree-sitter grammar folder and I see that injections.scm, etc., exist in its queries folder.

I see that the grammar, including the queries folder has been correctly checked out and is in runtime/grammars.

I’m confused.

I check out the documentation on adding languages and notice the note about how queries have to be added manually to runtime/queries.

I’m left wondering why running hx --grammar build my_lang didn’t just copy my queries folder from my tree-sitter grammar to runtime/queries for me.

Suggested implementation

Running hx --grammar build my_lang should copy the queries directory from the tree-sitter grammar (if one exists) to runtime/queries as part of the build/installation process.

[archseer]

I don't think that's a good idea since this only makes sense when you're building from source (where it's OK to have some extra steps -- you have to set up your runtime path. I actually recommend setting the env variable instead of copying or symlinking to the user config dir). Distributions package using the exact same commands except the runtime is stored in /usr/lib or similar.

[aral]

@archseer Does this mean that if I was running a distribution build, and I had, in my ~/.config/helix/languages.toml:

[[language]]
name="my_lang"
# …

[[grammar]]
name="my_lang
#…

And I ran:

hx --grammar fetch my_lang
hx --grammar build my_lang

The queries in the grammar from the git repository would automatically be used by Helix? Without having to manually create them in runtime/queries?

I guess I’m confused about why it would behave differently between when installed from a distribution and when built from source but I know very little of the inner workings of Helix Editor so it could be that I’m missing something fundamental here :)

[the-mikedavis]

The queries in the grammar from the git repository would automatically be used by Helix?

No, we don't use the queries from the grammar repos. #3020 (comment) is one reason but also:

• we use a custom set of scopes for syntax highlighting
• we have some injections like tree-sitter-comment which a grammar author may not include
• we have some custom queries for features other editors may not implement or may implement differently like textobjects or rainbow brackets (Rainbow tree-sitter matches 🌈 #2857)

It's also unreasonable on the grammar author for us to expect that any tree-sitter grammar we use has helix-specific queries, plus it would be a pain to update decentralized queries. The current state of the art is for grammar repositories to create queries that work with tree-sitter-cli's tree-sitter highlight subcommand. Editors then use custom queries based on those.

why it would behave differently between when installed from a distribution and when built from source

It depends on the distribution but typically they assemble a runtime directory in some /usr/lib directory rather than ~/.config/helix/runtime which is what the source will use by default.

In order to support syntax highlighting, we need the language and grammar configurations in languages.toml and also queries in the runtime directory under runtime/queries/<lang>/*.scm. For a language to be supported out of the box, that needs to be the languages.toml in the root of this repository and the runtime/ directory also in the root of repository. To support custom languages, you can add language and grammar configuration to the languages.toml in ~/.config/helix/languages.toml and put the queries in your runtime directory which is ~/.config/helix/runtime/ by default. But if you installed from a distribution, that may be somewhere else and you can consult the HELIX_RUNTIME environment variable.

[the-mikedavis]

All of that being said, it would be an improvement to bundle queries and grammars because queries depend on the version of the parser: query analysis will fail when old queries are used with a new parser that makes a breaking change like renaming or removing named nodes. I'll make a follow-up issue.

[the-mikedavis]

#3022