The big configuration and discovery rework

[ErinvanderVeen]

Topiary Configuration & Discovery Refactor

Wishlist

I want to generally unify Topiary's configuration, queries, and grammars. These is my wishlist:

1. Grammars in the configuration must be expressable in four ways:

   • Git repo (+ subdirectory)
   • Directory
   • Direct path to a compiled .so file

2. Queries in the configuration must be available in four ways:

   • Git repo (+ subdirectory)
   • Directory
   • Two direct paths to .scm files (formatting and injections)

3. Configuration File Resolution: The main configuration file (e.g., languages.ncl) must be discovered using an analogous order of precedence. However, because configurations are additive, Topiary should merge configurations from bottom to top (with higher precedence overriding lower precedence on a per-language basis):

   • A. Local Workspace / Project Directory
     Walk up from the current directory, stopping at the first .topiary/languages.ncl found (or at $HOME).
   • B. User Config Directory (XDG_CONFIG_HOME)
     Check XDG_CONFIG_HOME/topiary/languages.ncl (defaulting to ~/.config/topiary/languages.ncl). Note that this correctly targets .config unlike the data directories for queries/grammars.
   • C. System-Wide Config Directories (XDG_CONFIG_DIRS)
     Iterate through XDG_CONFIG_DIRS (defaulting to /etc/xdg), looking for topiary/languages.ncl.
   • D. Bundled Defaults (Base)
     The built-in configuration file shipped with Topiary forms the base of the merge strategy.

4. Auto-Discovery Formats & Directory Structure:
   While the languages.ncl configuration explicitly allows defining grammars/queries via Git repositories or source directories (which Topiary will fetch/compile), auto-discovery strictly looks for pre-compiled binaries and query text files.

   To ensure consistency, whenever Topiary checks a path during auto-discovery (whether .topiary/ locally, or XDG_DATA_HOME/topiary/ globally), it expects the following exact directory structure:

   topiary/ (or .topiary/)
   └── languages.ncl                  # Discovered during Configuration Resolution
   

[Xophmeister]

I like the auto-discovery unification 👍 That was always a bit of a bugbear of mine and I think your solution sounds good.

One nitpick: My experience is that environment variables usually take precedence over, in this case, local workspace/project directory defaults. We also have --options for some of these, which should be even higher priority, unless we opt to remove them in favour of this proposal.

[Xophmeister]

Ah, right...maybe it's a bit much, but would it be worth having "special" environment variables, just for Nix? Perhaps that's too messy. WDYT?

[ErinvanderVeen]

In nixpkgs/our flake we can always patch the source in some way, or we can enable a nix feature. But, ideally, I'd like topiary to work on NixOS and other Operating systems without any modification.

[prednaz]

the nix store paths of run time dependencies being passed at build time, is the idiomatic way, is it not?

[prednaz]

there is an even bigger problem than environment variable configuration yielding to project-local configurations. and that is environment variable configuration yielding to home directory configuration if i am not mistaken. so if i have configuration in my home directory, environment variable configuration is just never available to me anymore, is it?

[ErinvanderVeen]

the nix store paths of run time dependencies being passed at build time, is the idiomatic way, is it not?

Yes of course! We should just modify the builtin configuration.

[prednaz]

i am also very happy about finally treating grammars and queries the same. but did you consider removing auto discovery for queries rather than adding it for grammars, that is, making configuration files the only way to include grammars and queries? then we would not need to auto discover 3 different types of inputs, but configuration files only. "explicit is better than implicit."

[mkatychev]

DISCLAIMER: this is all theorycrafting for the future and somewhat tangential but something I do not think will go away without some planning

One problem that I've been dwelling on in maintaining grammars and queries for OpenSCAD is supporting both stable/2021 and latest grammar variants.

The initial approach I thought of is doing something akin to the Neovim ;; extends modeline to handle latest grammar by having fomatting_latest.scm have ;; extends formatting.scm at the top.
This can get messy quickly esp. if users want to mix and match grammars that can be mutually exclusive and now you would have to handle two different sources of truth again especially if, for instance, the default formatting.scm is incomplete by itself and must be composed to mutually exclusive variants (SQL grammars are a good example).

Pointing to a directory for grammars is definitely a good idea but we should be careful to assume that we have all the information to resolve a user's needs, ideally something akin to this:

{
openscad = {
      extensions | default = ["scad"],
      grammar.source | default = {
        git = {
          git = "https://github.com/openscad/tree-sitter-openscad.git",
          rev = "acc196e969a169cadd8b7f8d9f81ff2d30e3e253",
          nixHash = "sha256-x6fU1yPhYfoXemjswk+yRHW+c5V6nJgesK5tunYE7MI=",
        },
      },
      query.formatter.source = { 
         git = grammar.source.git, # reuse grammar.source stuff here
         # similar to .grammar.source.git.subdir
         # can be single path or multiple, if list then logic should function like `;; extends`
         path = ["./queries/formatting.scm", "./queries/fomatting_latest.scm" ] # can be 
    },
}

I realize this is way out of scope of the discussion but I feel what we decide here will colour our approach to future capabilities.

[ErinvanderVeen]

users want to install a bunch of different grammars/queries on the local fs

How do you foresee this happening, we currently don't have a repository/index for known external grammars. I think something like nvim-treesitter's setup would be nice: require('nvim-treesitter').install({...}) but we would need the index first and even then, the oneliner grammars should all go into the cache dir.

When a user would call something like topiary config <lang> --resolve, they should be getting a nickel record that shows the exact files that are on their system as a way to help them make sense where everything is explicitly. Whether we call directories or git sources, they should always end up pointing to an explicit file for the grammar, formatters, or injections.

Ultimately, I agree with your stance here, which is why I've removed the discovery from the proposal entirely.

[ErinvanderVeen]

DISCLAIMER: this is all theorycrafting for the future and somewhat tangential but something I do not think will go away without some planning

One problem that I've been dwelling on in maintaining grammars and queries for OpenSCAD is supporting both stable/2021 and latest grammar variants.

The initial approach I thought of is doing something akin to the Neovim ;; extends modeline to handle latest grammar by having fomatting_latest.scm have ;; extends formatting.scm at the top. This can get messy quickly esp. if users want to mix and match grammars that can be mutually exclusive and now you would have to handle two different sources of truth again especially if, for instance, the default formatting.scm is incomplete by itself and must be composed to mutually exclusive variants (SQL grammars are a good example).

Pointing to a directory for grammars is definitely a good idea but we should be careful to assume that we have all the information to resolve a user's needs, ideally something akin to this:

{
openscad = {
      extensions | default = ["scad"],
      grammar.source | default = {
        git = {
          git = "https://github.com/openscad/tree-sitter-openscad.git",
          rev = "acc196e969a169cadd8b7f8d9f81ff2d30e3e253",
          nixHash = "sha256-x6fU1yPhYfoXemjswk+yRHW+c5V6nJgesK5tunYE7MI=",
        },
      },
      query.formatter.source = { 
         git = grammar.source.git, # reuse grammar.source stuff here
         # similar to .grammar.source.git.subdir
         # can be single path or multiple, if list then logic should function like `;; extends`
         path = ["./queries/formatting.scm", "./queries/fomatting_latest.scm" ] # can be 
    },
}

I realize this is way out of scope of the discussion but I feel what we decide here will colour our approach to future capabilities.

I believe this should be handled upstream (if it isn't already). I.e. the tree-sitter library should have support for this.

[mkatychev]

Which section? the https://github.com/openscad/tree-sitter-openscad may be a poor choice of URL, for the example, we assume one git source has both grammar and formatter query sources (like https://github.com/TLATER/tree-sitter-sieve)

[ErinvanderVeen]

Imports. After all, it's just a version of scheme. I don't actually know if they already can, I don't assume so. Either way, that would be the best solution.