idea: rudimentary package management built into tree-sitter lib

[llllvvuu]

Distributing pre-generated parse tables uses much more bandwidth and hosting resources than the grammar and scanner files themselves.

Currently, the "clone and build" distribution method expects clients to have the tree-sitter CLI as well as a C toolchain. We could either remove all dependencies by distributing .wasm/.so (#730), or conversely expect a JS runtime (#2403) and greatly reduce download sizes (this would have savings increasing in the number of languages, whereas the cost is fixed). Or we could support both.

Rudimentary package management would contribute:

• A spec for parser distribution formats (analogous to .deb)
  • langname-src-langversion-abiversion.tar.gz - smallest
    • Contents: grammar.js, scanner.o (not technically "source", but maybe opens up to people writing in Zig/etc; must have no external dependencies), maybe *.scm
  • langname-wasm-langversion-abiversion.tar.gz - most portable without needing build toolchain
  • langname-arch-triple-langversion-abiversion.tar.gz - fastest without needing build toolchain
  • No format would include parser.c - it's even bigger than .so while requiring external dependency to build
  • But could a JSON-based distribution fall on the efficient frontier? In https://github.com/Julian/tree-sitter-lean, grammar.json is >100x smaller than lean.so and >500x smaller than parser.c
  • One challenge I could see with the src and json formats is that the tree-sitter runtime library is in C, but the generation stuff is in Rust?
• A common install prefix and local resolution scheme ($TS_PARSER_PATH/$TS_MAJOR_VERSION)
• A reference implementation / library that editors/etc can use

No ambition for a standard registry yet.

Very rough sketch

API

enum InstallFormat { SOURCE, WASM, NATIVE };
enum InstalledFormat { WASM, NATIVE };

bool ts__can_install_from_source(); // requires JS runtime and C toolchain
bool ts__can_install_from_url(); // requires some http lib
InstallStatus ts__install_from_file(
  InstallFormat format,
  char* fullname, // treesitter-python-abcdef00
  char* alias, // python
  char* filename, // path/to.tar.gz
  bool overwrite, // maybe more than 2 possible behaviors here
);
InstallStatus ts__install_from_url(
  InstallFormat format,
  char* fullname,
  char* alias,
  char* url,
  char* sha256,
  bool overwrite,
);
UninstallStatus ts__uninstall_alias(InstalledFormat format, char* alias);
UninstallStatus ts__uninstall_fullname(InstalledFormat format, char* fullname);
TSLanguage* ts__resolve_alias(InstalledFormat format, char* alias);
TSLanguage* ts__resolve_fullname(InstalledFormat format, char* fullname);

CLI

tree-sitter install [--source|--wasm|--native] <FULL_NAME> <ALIAS> <URL_OR_FILENAME>
tree-sitter uninstall [--wasm|--native] <FULL_NAME_OR_ALIAS>
tree-sitter which <ALIAS | FULL_NAME>

Editor/Plugin Usage

• Editor handles mapping of their internal filetype names to tree-sitter install aliases
• Editor handles running (if configured) ts__install_from_url whenever ts__resolve_alias returns NULL
• Editor provides :TSInstall/:TSForceInstall wrapping ts__install_from_url, supplying lookup table from language name to URL
• Editor uses resolved TSLanguage much like how they are using now. The tree-sitter library's resolution mechanism will only return TSLanguages that it's compatible with.
• If e.g. python is too generic an alias for a global install, an alias like nvim-python or helix-python could be used
• Bundling parsers could be done like how Helix is doing now, but could also be done by bundling source .tar.gz and inflating on first startup (prompting QuickJS/tinycc download if no Node.js/gcc/clang found).

[clason]

conversely expect a JS runtime (#2403)

Unacceptable for Neovim.

(I applaud your enthusiasm, but it would be a good idea to spend some time learning about the landscape and previous discussions on this topic -- both of which are quite complex -- first, to save everybody some time.)

The main thing is binary distribution for a (very!) wide range of platforms and architectures (and hosts), ideally without requiring any external toolchain locally. (The "editor usage" is a red herring; that part is already pretty much figured out.)

There's basically three choices:

1. require a toolchain (the current way -- this is not as horrible as you make it sound, since the parser.c compresses very well);
2. distribute binaries for a handful of "common platform/architecture" pairs (discussed on and off, but would leave out too many platforms that have Tier 1 support in, e.g., Neovim);
3. distribute a fully portable "binary" (which these days means WASM).

[clason]

just "check if this file is compatible", which has been discussed before

Without any resolution. The question of tree-sitter parser versioning has been open for some time -- again, due to the different requirements of "static" vs. "dynamic" consumers. That is critical issue; once that is resolved, the rest is paint by numbers.

[llllvvuu]

again, due to the different requirements of "static" vs. "dynamic" consumers

Intelligence w.r.t. dynamic loading is what I had in mind with the ts__resolve_alias sketch. Essentially if you've installed /ts-v2/lang-v3/{parser.so, query.scm} and /ts-v3/lang-v4/{parser.so, query.scm} then treesitter-lib would have great context in picking the right one to load (or returning NULL if none suitable). And the remote would have to have a similar tagging scheme.

nvim-treesitter already hits me with downloads whenever I up/downgrade nvim-treesitter, although it deletes versions I've already downloaded and should in theory also correspond with some mashup of Neovim up/downgrades and nvim-treesitter queries, rather than just nvim-treesitter up/downgrades; more specifically with tree-sitter lib up/downgrades. tree-sitter lib taking over some of the logic could just know that.

(even more true if nvim-treesitter/nvim-treesitter#3944 pans out)

But it wasn't my finest design, just meant to spark ideas.

So tl;dr the second half of the OP idea was exploring centralizing/upstreaming any of the TSInstall / Language:add logic (if not, then indeed there is nothing new over nvim-treesitter/nvim-treesitter#3944)

[clason]

None of this is a technical issue, though; it's a strategic design issue. (Nevertheless, I would recommend to invest some time learning about the actual tech, as

treesitter-lib would have great context in picking the right one to load

is just not how it works, and

nvim-treesitter already hits me with downloads whenever I up/downgrade nvim-treesitter, although it overwrites versions I've already downloaded

also indicates that something is going wrong on your end, as that is not expected behavior. (Is it Nix? It's always Nix...))

[llllvvuu]

also indicates that something is going wrong on your end, as that is not expected behavior. (Is it Nix? It's always Nix...))

I think it's just:

NOTE: This plugin is only guaranteed to work with specific versions of language parsers (as specified in the lockfile.json). When upgrading the plugin, you must make sure that all installed parsers are updated to the latest version via :TSUpdate. It is strongly recommended to automate this

is just not how it works

Can the same constants not be used to search the filesystem?

tree-sitter/lib/include/tree_sitter/api.h

Lines 169 to 179 in 2cbfcbc

	/**
	* Set the language that the parser should use for parsing.
	*
	* Returns a boolean indicating whether or not the language was successfully
	* assigned. True means assignment succeeded. False means there was a version
	* mismatch: the language was generated with an incompatible version of the
	* Tree-sitter CLI. Check the language's version using [`ts_language_version`]
	* and compare it to this library's [`TREE_SITTER_LANGUAGE_VERSION`] and
	* [`TREE_SITTER_MIN_COMPATIBLE_LANGUAGE_VERSION`] constants.
	*/
	bool ts_parser_set_language(TSParser *self, const TSLanguage *language);

tree-sitter/lib/include/tree_sitter/api.h

Lines 21 to 33 in 2cbfcbc

	* The latest ABI version that is supported by the current version of the
	* library. When Languages are generated by the Tree-sitter CLI, they are
	* assigned an ABI version number that corresponds to the current CLI version.
	* The Tree-sitter library is generally backwards-compatible with languages
	* generated using older CLI versions, but is not forwards-compatible.
	*/
	#define TREE_SITTER_LANGUAGE_VERSION 14
	/**
	* The earliest ABI version that is supported by the current version of the
	* library.
	*/
	#define TREE_SITTER_MIN_COMPATIBLE_LANGUAGE_VERSION 13

("generally backwards-compatible" "earliest ABI version" 🤔)

Like if you change the directory from parsers/ to parsers/v14/ and throw the *.scms in there too (parsers/v14/python-v6/*.scm) there should be no issues right?

[clason]

This is not the channel to answer such basic questions. Feel free to ask on Matrix.

(TL;DR: This is the consumer's responsibility. And the ABI is not the issue here.)