Split command identity into a join key and a signature (fixes #170, fixes #188)

[typeless]

Command identity was one SHA-256 doing two jobs that want opposite things. As the cross-build join key it wants to stay stable so a command keeps its history; as the change detector it wants to move the moment anything affecting the output changes. Fused, it could only do the second.

This started as #170's suggested fix — an adjacent_find to make identity collisions loud — and became the split, because enforcement kept revealing that the invariant being enforced was the wrong one.

commit
1ee5c0b	Reject two commands sharing one identity — fixes #170
90a56b1	Distinguish dep-scan commands by parent; exempt empty renders
c08c7ef	Split identity into key and signature — fixes #188
17bf5a2	File-granular stale removal; one join rule for all three directions

What was wrong

Editing only a rule's recipe — same inputs, same outputs — made it a different rule:

: in.txt |> cp %f %o |> out.txt        →        : in.txt |> cat %f > %o |> out.txt

New command:
Removed stale: out.txt        ← the artifact is deleted
Removed command:

Every consumer that joins by identity (implicit-dep routing, input-set reconciliation, stale-output removal, carried implicit edges, out-of-scope merge) loses the command's history with it. And because the join key is the identity function's output, every improvement to that function invalidated all joins globally — v11, v15, v16, v17, v18 each forced a full rebuild.

Upstream already splits them

tup/src/tup/parser.c:3573:

/* If we already have our command string in the db, then use that.
 * Otherwise, we try to find an existing command of a different
 * name that points to the output files we are trying to create. */

find_existing_command walks the rule's outputs and takes each one's incoming link. When that join lands on a command with a different string, tup calls tup_db_set_name and sets command_modified — the node survives, the string is the change signal. putup had fused what upstream keeps apart.

The split

key — which rule this is. A command that produces files is named by the files: output ownership is already unique among guard-satisfied commands, enforced where the output edge is created. Output-less commands have nothing else to be named by and fall back to a textual key of (text, dir, parent key).

signature — what it will do: rendered text, source dir, and the values of the vars it depends on. Compared only after a join succeeds.

The load-bearing equivalence — "identity equality ⟺ same command", never proven and violated in five recorded instances (#167, #171, #177/#178, #170, and the dep-scan case) — is replaced by an invariant the graph builder enforces.

Result on the reproducer: Changed command. The artifact is never deleted.

It also got faster: the key is the output path, which both graph and index already hold as edges, so the join needs no hash. Command-index time 0.5ms → 0.0ms on putup's own graph; 3.5ms → 1.6ms on a 5000-command one.

INDEX_VERSION 18 → 19; RawCommandEntry 48 → 80 bytes.

Two review passes, four defects, all mine

CI caught two. A dep-scan command is generated per compile with the parent's -o dropped, so two compiles of one source with equal flags render byte-identical scans — GCC does that, and my duplicate-check rejected it. Not an exemption: the discriminator was genuinely missing from the hash, which is #170's defect one level down. Separately, the configure pass evaluates before tup.config exists, so rules gated on unset config vars render alike; the check now sits out that pass entirely.

An adversarial review (Fable) caught two more, both blockers, both reproduced against a main-built binary:

1. Dropped outputs leaked permanently. remove_stale_outputs asked "did this command survive" and skipped the entry when it had. Sound while the join was strict; broken once it was forgiving. A rule that drops one of two outputs joins through the survivor, and the dropped file — absent from the new index, so its ownership record is gone — is never deleted by any later build. Staleness is a property of a file: an output with no live producer is stale, whoever used to own it. Now implemented that way.

2. The check bricked configure. My empty-render exemption covered only fully empty text; ./run @(A) and ./run @(B) both render "./run ", so a legal project could not be configured at all — and configure is the step that creates the file distinguishing them.

The two smaller findings shared one cause: the join rule existed in three ad-hoc copies (graph→index, index→graph, index→index) that had drifted apart, so the directions disagreed about whether a command survived. All three now go through one CommandLookup/CommandAddress pair.

Verification

make test: 142392 assertions, 634 cases, 32 e2e shards. make tidy, make iwyu clean. All 12 CI checks green.

Behaviour checked by execution, each before and after: recipe edit keeps the artifact and reports Changed command; dropping one of two outputs removes exactly that file; a deleted rule removes its output while its neighbour's survives; a renamed output is a new command with the old artifact removed; an exported-var change re-runs without deleting; scoped and out-of-tree builds unaffected.

Real projects: GCC BSP configures, parses (24 Tupfiles, 1853 commands) and builds; busybox parses (31 Tupfiles, 293 commands); coverage configure completes.

Not claimed

• Fable's fourth finding (preserve_old_implicit_edges / merge_out_of_scope_commands matching on bare key) was latent — no reproducer exists. The cause is fixed anyway by collapsing the three joins, but that part is reasoned, not demonstrated.
• The index has no single definition of a command's recorded state #189, Group-member removal is not routed: reconcile_input_set walks one hop, group membership needs two #169, remove_stale_outputs runs after all detection, so its own deletions are never routed to consumers #172 and A failed command is forgotten: the next build reports "Nothing to do" and exits 0 #187 are untouched. This fixes the join; The index has no single definition of a command's recorded state #189 is about the twelve hand-written record/compare/route paths, a separate root cause.
• metrics.command_expansions is printed by --stat and never incremented anywhere — noted, not fixed.

🤖 Generated with Claude Code

https://claude.ai/code/session_01TLob4Ef3qyMd3Dnp9q3XmA

[github-actions]

PR metrics

Performance (gcc example, Linux)

Workload	Instructions	CPU time	Page faults	D1 miss	LL miss	Wall	Peak RSS
parse	1075 M	0.2 s	14.2 k (+0.7%)	0.6%	0%	0.217 s	31.2 MB (+0.4MB)
dry-run	1880 M (+4.9%)	0.29 s	16.6 k	0.7%	0%	0.28 s	40.2 MB

Deterministic signals: instructions (cachegrind-simulated instruction reads — exact across runs, no PMU needed), page faults, peak RSS, and the cachegrind D1/LL miss rates. CPU time is user+sys from time(1).

Internal statistics (gcc example, up-to-date dry run)

Metric	Value
Tupfiles parsed	24
Commands	3545
Commands scheduled	0
Files checked	5818
Files changed	0
Files in index	6087
Graph edges	367913
Index size (bytes)	7378282 (+1.6%)
Implicit deps	330624
Hash computations	0
Hashes skipped (stat cache)	5818
Stat calls	5869
Parse time (ms)	202.5
Total time (ms)	265.7
Runner CPU	Intel(R) Xeon(R) 6973P-C

Counters from putup -n --stat on the fully-built gcc example (up-to-date dry run): deterministic work measures — a jump in commands scheduled, hash computations, or stat calls is a real behavior change, not noise. Timings are the minimum over repeated runs, compared only against a baseline from the same CPU model; the counters are the regression signal.
Timing deltas suppressed: baseline ran on different hardware (Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz).

Binary size (Linux)

Binary	.text	.data	.bss	File
putup	514.3 KB (+1.9%)	1.6 KB	98.7 KB	609.6 KB (+2.1%)

Test coverage (lines)

Overall	Median file	Min file	Max file
86.7% (+0.1pp)	95% (+0.1pp)	0.0% src/platform/path-posix.cpp	100.0% include/pup/core/arena.hpp

103 files · 15157/17488 lines covered

Deltas vs main@b3ffd393f.

Updated for 17bf5a2