Skip to content
This repository was archived by the owner on Sep 1, 2025. It is now read-only.
This repository was archived by the owner on Sep 1, 2025. It is now read-only.

Makes breaks when provenances are too large #1128

@dsalaza4

Description

@dsalaza4

When running jobs that have huge provenance files, the CLI function _nix_hashes receives too many arguments and breaks due to ARG_MAX limitations (depending on the OS).

Here is where the bug happens

makes/src/cli/main/cli.py

Lines 329 to 340 in 2244b2b

def _nix_hashes(*paths: str) -> List[str]:
cmd = [
f"{__NIX_STABLE__}/bin/nix-store",
"--query",
"--hash",
*paths,
]
out, stdout, _ = _run_outputs(cmd, stderr=None)
if out != 0:
raise SystemExit(out)
return stdout.decode().splitlines()

Specifically in the *paths argument provided to the shell (line 334).

Let's make _nix_hashes receive bytes from stdin and then call nix-store using xargs so it does not hit the ARG_MAX limit.

I found this bug by running m gitlab:fluidattacks/universe@trunk /docs/generate/criteria, as it currently depends on several hundred derivations for building the entire criteria section of the website.

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions