-
-
Notifications
You must be signed in to change notification settings - Fork 2.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
replace submodules with flags for fetching and building grammars #1659
replace submodules with flags for fetching and building grammars #1659
Commits on Mar 9, 2022
-
The submodules system is being replaced with a command-line flag hx --fetch-grammars Which shallow-clones grammar repositories at the given revision and hx --build-grammars For building grammars separate of the initial compilation of helix. Why remove submodules? * Cloning helix in general takes a long time because of the submodules, especially when the submodules are not fetched as shallow * Packaging is consistently painful no matter the package-manager * It is quite difficult to devise a scheme where users can declare a desired set of grammars and implement it with submodules This commit fully removes the existing tree-sitter submodules from the tree (as well as the .gitmodules file which is no longer used).
Configuration menu - View commit details
-
Copy full SHA for 9214395 - Browse repository at this point
Copy the full SHA 9214395View commit details -
add tree-sitter sources to languages.toml
Here we add syntax to the languages.toml languge [[grammar]] name = "<name>" source = { .. } Which can be used to specify a tree-sitter grammar separately of the language that defines it, and we make this distinction for two reasons: * In later commits, we will separate this code from helix-core and bring it to a new helix-loader crate. Using separate schemas for language and grammar configurations allows for a nice divide between the types needed to be declared in helix-loader and in helix-core/syntax * Two different languages may use the same grammar. This is currently the case with llvm-mir-yaml and yaml. We could accomplish a config that works for this with just `[[languages]]`, but it gets a bit dicey with languages depending on one another. If you enable llvm-mir-yaml and disable yaml, does helix still need to fetch and build tree-sitter-yaml? It could be a matter of interpretation.
Configuration menu - View commit details
-
Copy full SHA for 1f65862 - Browse repository at this point
Copy the full SHA 1f65862View commit details -
migrate helix-syntax crate into helix-core and helix-term
helix-syntax mostly existed for the sake of the build task which checks and compiles the submodules. Since we won't be relying on that process anymore, it doesn't end up making much sense to have a very thin crate just for some functions that we could port to helix-core. The remaining build-related code is moved to helix-term which will be able to provide grammar builds through the --build-grammars CLI flag.
Configuration menu - View commit details
-
Copy full SHA for fbb88e5 - Browse repository at this point
Copy the full SHA fbb88e5View commit details -
rename tree_sitter_library in LanguageConfig to 'grammar'
This is not strictly speaking necessary. tree_sitter_library was used by just one grammar: llvm-mir-yaml, which uses the yaml grammar. This will make the language more consistent, though. Each language can explicitly say that they use Some(grammar), defaulting when None to the grammar that has a grammar_id matching the language's language_id.
Configuration menu - View commit details
-
Copy full SHA for c824ad4 - Browse repository at this point
Copy the full SHA c824ad4View commit details -
Configuration menu - View commit details
-
Copy full SHA for 77e7945 - Browse repository at this point
Copy the full SHA 77e7945View commit details -
implement build_grammars and fetch_grammars
build_grammars adapts the functionality that previously came from helix-syntax to be used at runtime from the command line flags. fetch_grammars wraps command-line git to perform the same actions previously done in the scripts in helix-editor#1560.
Configuration menu - View commit details
-
Copy full SHA for 8dc1040 - Browse repository at this point
Copy the full SHA 8dc1040View commit details -
Configuration menu - View commit details
-
Copy full SHA for c994379 - Browse repository at this point
Copy the full SHA c994379View commit details -
Configuration menu - View commit details
-
Copy full SHA for 333a591 - Browse repository at this point
Copy the full SHA 333a591View commit details -
add 'use-grammars' to languages.toml
The vision with 'use-grammars' is to allow the long-requested feature of being able to declare your own set of grammars that you would like. A simple schema with only/except grammar names controls the list of grammars that is fetched and built. It does not (yet) control which grammars may be loaded at runtime if they already exist.
Configuration menu - View commit details
-
Copy full SHA for c273cc5 - Browse repository at this point
Copy the full SHA c273cc5View commit details -
migrate grammar fetching/building code into helix-loader crate
This is a rather large refactor that moves most of the code for loading, fetching, and building grammars into a new helix-loader module. This works well with the [[grammars]] syntax for languages.toml defined earlier: we only have to depend on the types for GrammarConfiguration in helix-loader and can leave all the [[language]] entries for helix-core.
Configuration menu - View commit details
-
Copy full SHA for 140c6da - Browse repository at this point
Copy the full SHA 140c6daView commit details -
Configuration menu - View commit details
-
Copy full SHA for 7b7143f - Browse repository at this point
Copy the full SHA 7b7143fView commit details -
Configuration menu - View commit details
-
Copy full SHA for b733525 - Browse repository at this point
Copy the full SHA b733525View commit details -
only fetch git-sourced grammars
This is a bit of a micro-optimization: in the current setup we waste a thread in the pool for a local grammar only to println! a message saying we're skipping fetching because it's a local grammar.
Configuration menu - View commit details
-
Copy full SHA for 3a21d61 - Browse repository at this point
Copy the full SHA 3a21d61View commit details -
fetch and compile tree-sitter grammars in helix-term build
This restores much of the behavior that existed before this PR: helix will build the grammars when compiling. The difference is that now fetching is also done during the build phase and is done much more quickly - both shallow and in parallel.
Configuration menu - View commit details
-
Copy full SHA for c2b6a0b - Browse repository at this point
Copy the full SHA c2b6a0bView commit details -
fetch and build grammars with nix in flake
This commit replaces the out-of-date builder in the flake which relied on submodules for fetching and the compiler for building. Now we disable fetching and building explicitly with the environment variable and then use builtins.fetchGit and a derivation mostly derived from upstream to compile the grammars. Anecdotally, this is still quite slow as builtins.fetchGit does not seem to do shallow clones. I'm not sure I see a way around it though without recording sha256s, which seems cumbersome.
Configuration menu - View commit details
-
Copy full SHA for 5b4aec7 - Browse repository at this point
Copy the full SHA 5b4aec7View commit details -
rename '--fetch/build-grammars' flags into '--grammar fetch/build'
The old flags were a bit long. --grammar is also aliased to -g to make it even easier.
Configuration menu - View commit details
-
Copy full SHA for c05b6b3 - Browse repository at this point
Copy the full SHA c05b6b3View commit details
Commits on Mar 10, 2022
-
flake: use builtins.fetchTree to shallow-clone grammar repos
Here we perform a shallow fetch using builtins.fetchTree. In order to make this work, we need to specify the `ref' for any repository that doesn't have `master' as its default branch (I'm not sure why this limitation exists since we don't need this when performing the shallow fetch in `--grammar build') This `ref' field is ignored by helix, so I have left it undocumented for now, but I could be open to documenting it.
Configuration menu - View commit details
-
Copy full SHA for af0383d - Browse repository at this point
Copy the full SHA af0383dView commit details -
update revision for tree-sitter-rescript
Looks like this was rebased a few hours ago and now the 789a171 revision no longer exists.
Configuration menu - View commit details
-
Copy full SHA for d0f4646 - Browse repository at this point
Copy the full SHA d0f4646View commit details