Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

replace submodules with flags for fetching and building grammars #1659

Merged
merged 18 commits into from
Mar 10, 2022

Commits on Mar 9, 2022

  1. remove all submodules

    The submodules system is being replaced with a command-line flag
    
        hx --fetch-grammars
    
    Which shallow-clones grammar repositories at the given revision and
    
        hx --build-grammars
    
    For building grammars separate of the initial compilation of helix.
    
    Why remove submodules?
    
    * Cloning helix in general takes a long time because of the submodules,
      especially when the submodules are not fetched as shallow
    * Packaging is consistently painful no matter the package-manager
    * It is quite difficult to devise a scheme where users can declare
      a desired set of grammars and implement it with submodules
    
    This commit fully removes the existing tree-sitter submodules from
    the tree (as well as the .gitmodules file which is no longer used).
    the-mikedavis committed Mar 9, 2022
    Configuration menu
    Copy the full SHA
    9214395 View commit details
    Browse the repository at this point in the history
  2. add tree-sitter sources to languages.toml

    Here we add syntax to the languages.toml languge
    
        [[grammar]]
        name = "<name>"
        source = { .. }
    
    Which can be used to specify a tree-sitter grammar separately of
    the language that defines it, and we make this distinction for
    two reasons:
    
    * In later commits, we will separate this code from helix-core
      and bring it to a new helix-loader crate. Using separate schemas
      for language and grammar configurations allows for a nice divide
      between the types needed to be declared in helix-loader and in
      helix-core/syntax
    
    * Two different languages may use the same grammar. This is currently
      the case with llvm-mir-yaml and yaml. We could accomplish a config
      that works for this with just `[[languages]]`, but it gets a bit
      dicey with languages depending on one another. If you enable
      llvm-mir-yaml and disable yaml, does helix still need to fetch and
      build tree-sitter-yaml? It could be a matter of interpretation.
    the-mikedavis committed Mar 9, 2022
    Configuration menu
    Copy the full SHA
    1f65862 View commit details
    Browse the repository at this point in the history
  3. migrate helix-syntax crate into helix-core and helix-term

    helix-syntax mostly existed for the sake of the build task which
    checks and compiles the submodules. Since we won't be relying on
    that process anymore, it doesn't end up making much sense to have
    a very thin crate just for some functions that we could port to
    helix-core.
    
    The remaining build-related code is moved to helix-term which will
    be able to provide grammar builds through the --build-grammars CLI
    flag.
    the-mikedavis committed Mar 9, 2022
    Configuration menu
    Copy the full SHA
    fbb88e5 View commit details
    Browse the repository at this point in the history
  4. rename tree_sitter_library in LanguageConfig to 'grammar'

    This is not strictly speaking necessary. tree_sitter_library was used by
    just one grammar: llvm-mir-yaml, which uses the yaml grammar. This will
    make the language more consistent, though. Each language can explicitly
    say that they use Some(grammar), defaulting when None to the grammar that
    has a grammar_id matching the language's language_id.
    the-mikedavis committed Mar 9, 2022
    Configuration menu
    Copy the full SHA
    c824ad4 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    77e7945 View commit details
    Browse the repository at this point in the history
  6. implement build_grammars and fetch_grammars

    build_grammars adapts the functionality that previously came from
    helix-syntax to be used at runtime from the command line flags.
    
    fetch_grammars wraps command-line git to perform the same actions
    previously done in the scripts in helix-editor#1560.
    the-mikedavis committed Mar 9, 2022
    Configuration menu
    Copy the full SHA
    8dc1040 View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    c994379 View commit details
    Browse the repository at this point in the history
  8. Configuration menu
    Copy the full SHA
    333a591 View commit details
    Browse the repository at this point in the history
  9. add 'use-grammars' to languages.toml

    The vision with 'use-grammars' is to allow the long-requested feature
    of being able to declare your own set of grammars that you would like.
    A simple schema with only/except grammar names controls the list
    of grammars that is fetched and built. It does not (yet) control which
    grammars may be loaded at runtime if they already exist.
    the-mikedavis committed Mar 9, 2022
    Configuration menu
    Copy the full SHA
    c273cc5 View commit details
    Browse the repository at this point in the history
  10. migrate grammar fetching/building code into helix-loader crate

    This is a rather large refactor that moves most of the code for
    loading, fetching, and building grammars into a new helix-loader
    module. This works well with the [[grammars]] syntax for
    languages.toml defined earlier: we only have to depend on the types
    for GrammarConfiguration in helix-loader and can leave all the
    [[language]] entries for helix-core.
    the-mikedavis committed Mar 9, 2022
    Configuration menu
    Copy the full SHA
    140c6da View commit details
    Browse the repository at this point in the history
  11. fix context in error

    dead10ck authored and the-mikedavis committed Mar 9, 2022
    Configuration menu
    Copy the full SHA
    7b7143f View commit details
    Browse the repository at this point in the history
  12. shallow clone

    dead10ck authored and the-mikedavis committed Mar 9, 2022
    Configuration menu
    Copy the full SHA
    b733525 View commit details
    Browse the repository at this point in the history
  13. only fetch git-sourced grammars

    This is a bit of a micro-optimization: in the current setup we waste
    a thread in the pool for a local grammar only to println! a message
    saying we're skipping fetching because it's a local grammar.
    the-mikedavis committed Mar 9, 2022
    Configuration menu
    Copy the full SHA
    3a21d61 View commit details
    Browse the repository at this point in the history
  14. fetch and compile tree-sitter grammars in helix-term build

    This restores much of the behavior that existed before this PR:
    helix will build the grammars when compiling. The difference is that
    now fetching is also done during the build phase and is done much
    more quickly - both shallow and in parallel.
    the-mikedavis committed Mar 9, 2022
    Configuration menu
    Copy the full SHA
    c2b6a0b View commit details
    Browse the repository at this point in the history
  15. fetch and build grammars with nix in flake

    This commit replaces the out-of-date builder in the flake which relied
    on submodules for fetching and the compiler for building. Now we
    disable fetching and building explicitly with the environment variable
    and then use builtins.fetchGit and a derivation mostly derived from
    upstream to compile the grammars.
    
    Anecdotally, this is still quite slow as builtins.fetchGit does not
    seem to do shallow clones. I'm not sure I see a way around it though
    without recording sha256s, which seems cumbersome.
    the-mikedavis committed Mar 9, 2022
    Configuration menu
    Copy the full SHA
    5b4aec7 View commit details
    Browse the repository at this point in the history
  16. rename '--fetch/build-grammars' flags into '--grammar fetch/build'

    The old flags were a bit long. --grammar is also aliased to -g to make
    it even easier.
    the-mikedavis committed Mar 9, 2022
    Configuration menu
    Copy the full SHA
    c05b6b3 View commit details
    Browse the repository at this point in the history

Commits on Mar 10, 2022

  1. flake: use builtins.fetchTree to shallow-clone grammar repos

    Here we perform a shallow fetch using builtins.fetchTree. In order
    to make this work, we need to specify the `ref' for any repository
    that doesn't have `master' as its default branch (I'm not sure why
    this limitation exists since we don't need this when performing
    the shallow fetch in `--grammar build')
    
    This `ref' field is ignored by helix, so I have left it undocumented
    for now, but I could be open to documenting it.
    the-mikedavis committed Mar 10, 2022
    Configuration menu
    Copy the full SHA
    af0383d View commit details
    Browse the repository at this point in the history
  2. update revision for tree-sitter-rescript

    Looks like this was rebased a few hours ago and now the 789a171
    revision no longer exists.
    the-mikedavis committed Mar 10, 2022
    Configuration menu
    Copy the full SHA
    d0f4646 View commit details
    Browse the repository at this point in the history