Auto-batching breaks down on a sufficiently large monorepo on Linux

[astei]

Disclaimer: I used Claude / Codex to help investigate this issue, but I wrote this summary myself.

I am trying to use hk with a very large monorepo (~110,000+ files) and noticed several problems with hk when trying it out. Everything worked great on macOS, but when I transferred over to our Linux CI environment, problems began to appear immediately.

Specifically, the Linux kernel has a per-argument size cap (MAX_ARG_STRLEN) of 128KiB by default, and this is in addition to the ARG_MAX headroom. Batching does not correctly account for this today, so a sufficiently large tree will cause hk run check <HOOK> --all to fail (either with exit code 7, E2BIG, or sh complaining about the argument list being too long).

This can be reproduced with a minimal hk.pkl:

amends "package://github.com/jdx/hk/releases/download/v1.50.0/hk@1.50.0#/Config.pkl"
hooks {
    ["check"] { steps { ["count"] { glob = List("**/*.txt"); check = "wc -c {{files}}" } } }
}

How do we resolve this?

I see two clear solutions, both of which are complementary:

1. hk should gain support for bypassing the shell, resolving to the appropriate executable on the PATH. hk could detect if a command actually needs to run in the shell (checking for characters with special meaning in a shell) and only spawning a shell then.
2. Regardless of that, on Linux, we have to cap batches that pass through the shell to 128KiB at maximum, in addition to the overall _SC_ARG_MAX size.

[jdx]

Confirmed—thanks for the detailed report. hk renders the full command and passes it as one sh -c argument, so Linux's per-string MAX_ARG_STRLEN applies independently of the aggregate ARG_MAX value. On a typical 4 KiB-page glibc system, that means the rendered command fails at 128 KiB even though hk's existing ARG_MAX / 2 threshold is roughly 1 MiB.

I also found a second issue in the batching implementation: it derived one file-count batch size from the first paths and reused it for all later chunks. A later chunk with longer paths could therefore still exceed the byte limit.

PR #1066 addresses both problems by:

• capping Linux shell-command arguments at 32 pages (minus the terminating NUL), in addition to the aggregate ARG_MAX margin;
• sizing every batch independently;
• returning a clear error if a single-file command cannot fit; and
• adding regression coverage above the Linux 128 KiB boundary.

I kept direct executable invocation out of this focused fix. It remains a useful complementary feature, but an explicit structured argv configuration would be safer than heuristically deciding whether an existing command string requires shell evaluation.

This comment was generated by Codex.

[astei]

Thanks @jdx! I would actually appreciate structured argv configuration as a feature as well, because the 128KiB cap does materially limit what I can do with hk on this particular repo.

[jdx]

@astei Implemented in PR #1067: #1067

The new form is explicit and backward-compatible:

check = new Command {
  argv = List("wc", "-c", "{{files}}")
}

argv[0] is resolved as the executable and run directly without a shell. Exact {{files}} and {{workspace_files}} entries expand into distinct raw arguments, so filenames retain their boundaries and shell metacharacters are not interpreted. Existing string commands and platform Script objects keep their current shell behavior.

Direct commands still auto-batch against the platform aggregate process limit, but they are no longer constrained by Linux’s 128 KiB per-string limit for the rendered sh -c command.

This comment was generated by Codex.