[feat] improve tree-sitter parsers search logic

[msva]

Problem

I've found that there are some tree-sitter force-enablements for some filetypes in master.
(actually, I was just triggered today by lua one, enforced on 1st January, and found help and query while investigated).

Unfortunatelly, we're on Gentoo build neovim without any bundles (and also fetching deps during build is against packaging guidelines), so we're on "provide all dependencies yourself" path.

It works for pretty all nvim dependencies, but has an issue with tree-sitter grammars.

We're installing it as /usr/${libdir}/libtree-sitter-${name}.so (and it seems pretty all tree-sitter using software that we have packaged (and it seems like even emacsers) are using tree-sitter parsers by those paths.

But NeoVim expects them as /usr/${libdir}/nvim/parser/${name}.so.
(Well, actually it expects them at &runtimepath/parser/${name}.so, but /usr/${libdir}/nvim, as far as I see, is the path your buildsystem uses to install them when buildtime fetching enabled).

So, as nvim can't find the grammars, it fails (even nvim --clean) on startup when opening corresponding files (at least, I reproduced on lua and help).

So, when we discussed about the ways of solving that issue, we decided to firstly ask if it is possible to add alternative search paths (and naming) on your side?
Maybe we can collaborate on PR for that, if you don't mind?

Although, I just found that alpine packages tree-sitter grammars at two path simultaneously: /usr/${libdir}/libtree-sitter-${name}.so (as we do) and /usr/${libdir}/tree-sitter/${name}.so, which seems like compatible with nvim's behaviour, and it will probably be enough to symlink /usr/${libdir}/tree-sitter to /usr/${libdir}/nvim/parser. What do you think about that way?

Thanks in advance!

Expected behavior

[not relevant]

[clason]

You can already use alternative search paths: just put them on runtimepath. Be mindful that you respect the ordering for the corresponding (compatible) queries.

The current defaults are finely tuned to support plugins and users overriding the bundled parsers and queries and will not be changed.

Officially, parsers (in the exact version and revision specified) are now part of the bundled runtime, and a build without them is considered broken.

[clason]

So, yes, the symlink way is the right way. But again, make sure that you put there the exact revision specified in our build script; queries expect a certain revision and are not guaranteed to be compatible with others. (This is not a theoretical issue.)

While we try to stick to tagged versions for our releases, nightly is free to make use of prerelease commits.

It's also not safe to assume that different editors target the same parser version (and this is impossible to check at runtime).

TL;DR: Neovim intentionally does not support using system-installed parsers since we have no control over them. This will not change until tree-sitter adds support for parser version and introspection.