How do I actually set up a hirearchy of configuration?

[dkowis]

I'm struggling with the PKL configuration resolution to set up a global "everyone has this ~/.config/hk/config.pkl" that has the hooks we want to have on everything all the time. And then adding a per-repo hk.pkl that adds more to it.

I don't think I need to do amends ~/.config/hk/config.pkl because that doesn't actually work, no shell expansion in the amends. And the documentation seems to mention that the ordered loading of things happens and merges them.

https://hk.jdx.dev/configuration.html#hkrc Yes, hkrc is deprecated, and the "global user config" uses the mentioned config.pkl.

I have a ~/.config/hk/config.pkl:

amends "package://github.com/jdx/hk/releases/download/v1.49.0/hk@1.49.0#/Config.pkl"
import "package://github.com/jdx/hk/releases/download/v1.49.0/hk@1.49.0#/Builtins.pkl"

local linters = new Mapping<String, Step> {
    ["alejandra"] = Builtins.alejandra
    ["google-java-format"] = Builtins.google_java_format

    ["byte-order-marker"] = Builtins.byte_order_marker
    ["check-added-large-files"] = Builtins.check_added_large_files
    ["check-case-conflict"] = Builtins.check_case_conflict
    ["check-executables-have-shebangs"] = Builtins.check_executables_have_shebangs
    ["check-merge-conflict"] = Builtins.check_merge_conflict
    ["check-symlinks"] = Builtins.check_symlinks
    ["detect-private-key"] = Builtins.detect_private_key
    ["mixed-line-ending"] = Builtins.mixed_line_ending
    ["newlines"] = Builtins.newlines
    ["trailing-whitespace"] = Builtins.trailing_whitespace
}

hooks {
    ["pre-commit"] {
        fix = true
        stash = "git"
        steps = linters
    }
    ["fix"] {
        fix = true
        steps = linters
    }
    ["check"] {
        steps = linters
    }
}

And then I want to have a repo config that adds more to it:

amends "package://github.com/jdx/hk/releases/download/v1.49.0/hk@1.49.0#/Config.pkl"
import "package://github.com/jdx/hk/releases/download/v1.49.0/hk@1.49.0#/Builtins.pkl"

// This is local only to this file
local my_linters = new Mapping<String, Step> {
    ["prettier"] = Builtins.prettier
    ["eslint"] = Builtins.eslint
    ["knip"] = Builtins.knip
}

hooks {
    ["pre-commit"] {
        fix = true
        stash = "git"
        steps = my_linters
    }
    ["fix"] {
        fix = true
        steps = my_linters
    }
    ["check"] {
        steps = my_linters
    }
}

I'll get random sporatic errors where hk fix will work but hk check does not work, there's gotta be something that I'm not getting in how the configuration is loaded and merged...

Would that merge in ways I expect so that all the linters are run? Or what's the right way to establish this? Or can this even happen and I'm thinking about it the wrong way?

[jdx]

Your mental model is correct: hk evaluates the global and project Pkl files independently, then merges them internally. You do not need to amends ~/.config/hk/config.pkl (and, as you noticed, Pkl does not perform shell expansion there).

With the two configs shown, the pre-commit, fix, and check hooks should each contain both sets of steps. Step names that exist only in the global config are added; step names that exist in both configs use the project definition.

One nuance: when both configs define the same hook, hk merges the step maps, but it does not deeply merge all hook-level properties. The project hook owns properties such as fix, stash, and hook environment. You repeat the relevant properties in this example, so that should not be a problem here.

The sporadic hk fix versus hk check errors do not look like an expected consequence of this merge. Could you share the exact error output, along with these results?

hk --version
hk check --all --plan
hk fix --all --plan

The two plans should make it immediately apparent whether this is config merging, file selection, or a particular builtin behaving differently in check mode.

This reply was generated by Codex.

[dkowis]

So there's some more context. I was running into what codex thinks are parallel execution and configuration valuation problems.

• mise x -- hk check: correct path, but your global hk utility built-ins run many nested hk util ... commands in parallel with jobs=14, and those intermittently trip over hk config/cache evaluation.

So to make those configuration red herrings go away, I have to serialize the operation of the util checks. like so:

• Yes. Add depends only between the global steps that shell out to hk util .... Do not chain repo tools like prettier, eslint, or knip.

In ~/.config/hk/config.pkl, change your utility steps to a dependency chain like this:

  local linters = new Mapping<String, Step> {
      ["alejandra"] = Builtins.alejandra
      ["google-java-format"] = Builtins.google_java_format

      ["byte-order-marker"] = Builtins.byte_order_marker

      ["check-added-large-files"] = (Builtins.check_added_large_files) {
          depends = List("byte-order-marker")
      }
      ["check-case-conflict"] = (Builtins.check_case_conflict) {
          depends = List("check-added-large-files")
      }
      ["check-executables-have-shebangs"] = (Builtins.check_executables_have_shebangs) {
          depends = List("check-case-conflict")
      }
      ["check-merge-conflict"] = (Builtins.check_merge_conflict) {
          depends = List("check-executables-have-shebangs")
      }
      ["check-symlinks"] = (Builtins.check_symlinks) {
          depends = List("check-merge-conflict")
      }
      ["detect-private-key"] = (Builtins.detect_private_key) {
          depends = List("check-symlinks")
      }
      ["mixed-line-ending"] = (Builtins.mixed_line_ending) {
          depends = List("detect-private-key")
      }
      ["newlines"] = (Builtins.newlines) {
          depends = List("mixed-line-ending")
      }
      ["trailing-whitespace"] = (Builtins.trailing_whitespace) {
          depends = List("newlines")
      }
  }

That info does vibe with the "random sporatic errors" being some kind of race condition, where the config.pkl is being evaluated lots of times, but I'm not super confident, it would line up with some of the things that happened when I set it up and the AI suggested that I set the job concurrency to 1, to keep it from barfing out random errors...

[dkowis]

with regards to the requested data:

❯ hk --version
hk 1.51.0

❯ hk check --all --plan
✔ Fetching all files in repo
Plan: check
Run type: check

  [parallel group] group_0
    ✓ prettier  (292 files matched)
    ✓ eslint  (228 files matched)
    ✓ knip  (231 files matched)
    ○ alejandra  (no files matched filters)
    ○ google-java-format  (no files matched filters)
    ✓ byte-order-marker  (360 files matched)
    ✓ check-added-large-files  (360 files matched)
    ✓ check-case-conflict  (360 files matched)
    ✓ check-executables-have-shebangs  (360 files matched)
    ✓ check-merge-conflict  (360 files matched)
    ✓ check-symlinks  (360 files matched)
    ✓ detect-private-key  (360 files matched)
    ✓ mixed-line-ending  (360 files matched)
    ✓ newlines  (360 files matched)
    ✓ trailing-whitespace  (360 files matched)

❯ hk fix --all --plan
✔ Fetching all files in repo
Plan: fix
Run type: fix

  [parallel group] group_0
    ✓ prettier  (292 files matched)
    ✓ eslint  (228 files matched)
    ✓ knip  (231 files matched)
    ○ alejandra  (no files matched filters)
    ○ google-java-format  (no files matched filters)
    ✓ byte-order-marker  (360 files matched)
    ✓ check-added-large-files  (360 files matched)
    ✓ check-case-conflict  (360 files matched)
    ✓ check-executables-have-shebangs  (360 files matched)
    ✓ check-merge-conflict  (360 files matched)
    ✓ check-symlinks  (360 files matched)
    ✓ detect-private-key  (360 files matched)
    ✓ mixed-line-ending  (360 files matched)
    ✓ newlines  (360 files matched)
    ✓ trailing-whitespace  (360 files matched)

So the merging did indeed happen, as I see things being defined in the plan, but the random crashes are I think from the utils all running at the same time and hitting some kind of gnarly thread bug on the configuration parsing...

[jdx]

I opened #1078 to make hk util skip project and global configuration loading entirely. That should remove the redundant parallel Pkl evaluations without requiring the utility steps to be chained.

@dkowis, would you be willing to build that branch and retry with normal concurrency and without the temporary depends chain? If it still fails, please share the exact error output so we can narrow down any remaining shared-cache issue.

This comment was generated by Codex.

[dkowis]

Sure I can give that a shot.

[dkowis]

I did:

[tools]
"cargo:https://github.com/jdx/hk" = { version = "rev:6e1a108a7dab597633a90eba26fa3020340168ab", crate = "hk" }

#hk = "1.51.0"

To run that version of hk directly, and the config loading race condition I could not get to show up again. Repeated runs show up reliably, so yeah I'd say that fixes it :D