fix(hamt): re-merge under-full subtrees so tree shape is canonical

[rmanibus]

Summary

• Make a HAMT's shape a function of its contents alone. Txn.delete now re-merges a subtree back into a leaf once it holds maxLeafSize entries or fewer, restoring the invariant buildNode already maintains on insertion: an internal node exists only where more than a leaf's worth of entries live beneath it.
• Budget the check so it cannot become the new problem: subtreeEntries abandons a subtree as soon as it has seen more entries than a leaf could hold.
• Add TestRootHashIsDeletionHistoryIndependent as the deletion counterpart to the existing order-independence test, plus a traversal-budget test and a regression pin for whole-directory deletion.
• Document the canonical-shape rule in docs/spec.md.

Closes #421.

Measurement first

The issue asked for a realistic measurement before committing to a change in a core data structure. That measurement changed the picture, and is worth reading before the diff:

workload	before	after
steady churn 1%/day, 90 days	1.00× — already canonical	unchanged
drop 25% of directories, then churn	1.00× — already canonical	unchanged
net shrink to ~10% over 90 rounds	11.91× (381 nodes vs 32)	1.00×
monotonic shrink (the issue's synthetic case)	8.00×	1.00×

The two cases I would have guessed were worst — steady churn and wholesale directory removal — were already exactly canonical. Affinity routing gives a directory a shared routing prefix, so deleting one empties whole subtrees, which the existing empty-slot and single-leaf rules already handle.

The damage comes from scattered deletion without replacement, and there it is worse than the issue estimated: 11.9× rather than the 5× the synthetic workload suggested. That run also wrote 6907 nodes where the fixed version writes 4268, so this reduces write amplification during the history, not only the final node count.

The two already-canonical workloads produce byte-identical node counts after the change (712 and 546), so nothing regressed to buy this.

Why not "sort the children"

Worth stating since it was the first instinct: children are already canonical, indexed by hash bits through the node bitmap, and insertion order already does not matter — TestRootHashIsOrderIndependent has pinned that all along. Deletion was the only asymmetry.

Cost

subtreeEntries stops once it has seen maxLeafSize + 1 entries, so a subtree too large to collapse is rejected after a handful of node loads. One delete in a 4000-entry tree reads 16 nodes; without the budget the same delete reads 532. TestCollapseDoesNotTraverseLargeSubtrees pins this and fails if the budget is removed — verified by removing it.

Compatibility

No format version bump, and no encoding change. What changes is which shapes get written, not how a shape is encoded. A leaf appearing mid-tree is already what a bulk build produces for any subtree small enough to fit in one.

Verified against a v1.18.0 binary in both directions rather than reasoned about:

• Old build, new tree. list, ls, check -read-data, restore, diff, and backup on top all work against a repository whose tree collapsed to a single leaf.
• New build, old tree. This build reads a tree an older one left non-canonical, and check -read-data reports it healthy.
• -ignore-empty-snapshot is unaffected. A no-change backup performs no deletions, so no collapse fires and the root is unchanged; the snapshot count stayed at 2.

A non-canonical tree written by an older build stays valid and becomes canonical the next time a deletion passes through it, which is the opportunistic-upgrade behaviour docs/compatibility.md asks for.

On the golden file: roothash_test.go warns that a changed root is a format change, not something to regenerate away. All four pinned roots are unchanged — that is the evidence the encoding is untouched. The golden gains one line because I added a scenario: the existing one named for collapse-on-delete leaves 60 entries, above maxLeafSize, so it never reached the new rule and the shape of a shrinking tree would have stayed unpinned.

Verification

• env GOCACHE=/tmp/cloudstic-gocache go test -count=1 ./...
• env GOCACHE=/tmp/cloudstic-gocache go test -race -count=1 ./internal/hamt ./internal/engine ./
• env GOCACHE=/tmp/cloudstic-gocache GOLANGCI_LINT_CACHE=/tmp/cloudstic-golangci-lint golangci-lint run ./...
• npx markdownlint-cli2 docs/spec.md
• Manual old-binary compatibility runs described above.

[codecov]

Codecov Report

❌ Patch coverage is 67.74194% with 10 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
internal/hamt/hamt.go	67.74%	5 Missing and 5 partials ⚠️

📢 Thoughts on this report? Let us know!