Performance

Performance

Status: measurement (Issue #27, 2026-09-28, develop d86ae3d, bench commit 77cb0dc on hub/issue-27), then bottleneck #1 fixed by Issue #28 (commit 59ff675 on hub/issue-28; see "After #28" below), then bottleneck #2 fixed by Issue #29 (commit da03f14 on hub/issue-29; see "After #29" below), then 1M ops measured by Issue #30 (see "Issue #30" below), then checkpoint decode and short-id index fixed by Issue #31 (see "Issue #31" below), then per-command overhead cut by Issue #32 (see "Issue #32" below), then on-disk format v1 (snapshots + batch commit records) by Issue #33 (see "Issue #33" below; D-026 status/record <= 1 s at 1M ops now met), then sync blob transfer by Issue #34 (see "Issue #34" below; D-030 100-replica run still partial, bound by per-file creation; pack format deferred by user decision). The targets are the Planner defaults from #27 and the user has not confirmed them yet. Cells marked ~ are partial observations and cells marked (extrapolated) were not run.

Issue #34: sync blob transfer, D-030 100 replicas (2026-09-29)

Branch hub/issue-34 (base develop 2c7dd10). Spec: Repository-Semantics A.2.8. Release, caffeinate -i, every run capped with timeout -s KILL. Data: hub-bench d030 (new case): a 1000-file base, R replicas copied with cp -R + hub reid, each records 10 rounds of 100 unique-content edits (1000 SetContent ops and 1000 unique blobs per replica; R=100 gives 102,010 ops and ~100k blobs). Star = replica k syncs with replica 0, k = 1..R-1, repeated until heads agree; ring = replica i syncs with i+1.

Profile before (develop 2c7dd10, star R=50, sync k~22, sample 6 s): rename(2) 512 samples (~62 %), F_FULLFSYNC 139, open 67, write 38, fsync 18, SHA-256 10. load also read and hashed every blob of both repos on every sync (RSS grew 31 -> 357 MB over k = 1..49). The peers genuinely lack the blobs they receive in a star (the delta never re-sent held blobs in this workload), so "send only missing" alone does not cut the star's volume; it does cut all-already-held and overlap cases to 0 bytes (tests).

Per-file cost on this Mac (APFS, python microbench, 20k files): rename 178-187 µs flat vs 167-183 µs with ab/ fan-out; 120-134 µs at 200-2000 files; create 137 µs; link 342 µs. Flat-dir size is not the cost, so no fan-out (A.2.8). The per-file creation cost is.

Fix (A.2.8): only lacking blobs travel (NeedBlobs step); held blobs are never read; received blobs are written directly to their final names (no temp, no rename), one directory flush + one barrier per receiving side.

Per-sync wall s (peak RSS MB), star, pass 1:

k	before, R=50	after, R=50	after, R=100	after, ring R=100
1	1.01 (31)	0.54 (29)	0.66 (33)	0.60 (31)
10	3.76 (92)	2.33 (96)	2.34 (99)	2.40 (92)
20	7.28 (167)	4.33 (164)	4.14 (173)	4.12 (165)
30	10.80 (234)	7.08 (241)	6.47 (253)	6.62 (233)
40	19.00 (307)	8.99 (274)	8.34 (308)	8.47 (303)
49/50	21.94 (357)	11.05 (351)	11.68 (380)	16.98 (332)
60	-	-	17.71 (438)	17.90 (436)
70	-	-	23.77 (444)	killed at cap
80	-	-	24.82 (453)	-
90	-	-	29.94 (584)	-
99	-	-	31.75 (642)	-
pass 1 total	550.9	295.6	1410.3	-

• R=50 star (after): converged in 2 passes, 98 syncs, 608 s (pass 2 286.8 s, slowest sync 14.7 s), head 81c51e35, 52,010 ops, same head and identical checked-out trees on all 50. Status at replica 0 0.29 s / 179 MB, record 10 edits 0.52 s. Before #34 pass 1 alone was 551 s.
• R=100 star: partial. Build 213 s; pass 1 1410 s; killed at the 2400 s cap during pass 2, after 28 pass-2 syncs (r001..r028; r099, the last of pass 1, already had everything). At the cap, 30/100 replicas (0, 1..28, 99) had replica 0's head 1491bce0 (102,010 ops, 100 replica ids) and the 29 checked-out trees were identical. A pass-2 sync of one more replica (r050, 49,000 ops + 49,000 blobs) took 15.0 s / 636 MB. At replica 0 (102k ops): status 0.33-0.35 s / ~390-400 MB, log 0.07 s / 261 MB, record 1 edit 0.15 s / 271 MB.
• R=100 ring: partial. Build 273 s, killed at the 900 s cap in pass 1 at k~69; per-sync times track the star's.
• Bottleneck now: per-file creation. After the fix, sample of a star sync: open 2667 samples (the direct create is the bulk; the sender's verified read and the receiver's existence probe are the rest), fsync 658, write 479, close 227; nothing else above 20. Full replication of R replicas × ~1000 blobs each writes ~R² × 1000 blob files (~10M at R=100, ~200 µs each), i.e. ~2000 s per topology on this machine; star pass 1 alone writes R²/2 × 1000. Deleting the R=100 star data (~3M files) took 17.7 min with rm -rf. The remedy is a pack format (many blobs per file per sync); deferred by user decision (#34). Sync RSS grows ~6 MB per 1000 ops (both op stores in memory), 642 MB at 100k.

Issue #33: format v1 snapshots, 1M ops, 100-replica re-measure (2026-09-29)

Branch hub/issue-33 (base develop 0aff2d9), final commit 3b3c04c. Spec: Repository-Semantics Appendix A.2. Release, caffeinate -i, every run capped with timeout -s KILL 900. Same data as #32: 100k = hub-bench ops --files 5000 --per 5000 (100,050 ops, built by the develop 0aff2d9 v0 binary), 1M = --files 10000 --per 10000 (1,000,100 ops, 98 checkpoints). Each size was copied (APFS clone), migrated by its first v1 open, then measured. 5 runs, median (min-max), wall s. "no snapshot" = HUB_NO_SNAPSHOT=1 (full replay). Stdout and stderr of status, log, ls -l and show <head> were asserted byte-identical with and without the snapshot at both sizes.

command	100k snapshot	100k no snapshot	1M snapshot	1M no snapshot	#32 1M	target	1M pass
ls (open only)	0.052 (0.050-0.052)	0.203	0.437 (0.430-0.440)	2.113	2.550	-	-
status clean	0.090 (0.089-0.093)	0.267	0.634 (0.623-0.646)	2.492	3.345	<= 1 s	yes
status 1 edit	0.097 (0.090-0.105)	0.266	0.669 (0.619-0.699)	2.520	3.345	<= 1 s	yes
record 1 edit	0.149 (0.147-0.151)	0.322	0.617 (0.611-0.633)	2.462	2.904	<= 1 s	yes
record -m	0.170 (0.168-0.173)	0.343	0.808 (0.801-0.817)	2.683	3.212	-	-
log	0.056 (0.054-0.058)	0.221	0.504 (0.497-0.514)	2.347	2.778	-	-
show head	0.063 (0.062-0.064)	0.230	0.512 (0.508-0.522)	2.314	2.761	-	-

• Migration v0 -> v1 (first open, includes validation, the v1 log rewrite, the switch and the first snapshot): 0.46 s at 100k, 4.04 s at 1M (4.4-5.1 s in earlier runs).
• Sizes at 1M: ops.log 195.7 MB (v0 195.7 MB + one commit record), snapshot 83.4 MB. 100k: 19.1 MB / 8.3 MB.
• The "record" rows run with a small tail after the snapshot (the policy writes a new snapshot only every max(4096, ops/16) ops), which is the normal case.

What the snapshot run spends at 1M (macOS sample, status). Open ~0.4 s: read + checksum + decode 83 MB, rebuild the op store from the applied log, re-read checkpoints. Then conflicts ~0.12 s, short-id index ~0.07 s, working-dir scan ~0.04 s.

Changes that got there, besides the snapshot:

• Causality uses a sorted id vector and CSR deps instead of hash maps, and Replica::conflicts passes log positions (conflict derivation at 1M: ~0.48 s -> ~0.12 s). Covered by the #28/#32 equivalence proptests (unchanged, 512 cases each).
• The CLI process no longer frees the replayed repository at exit (set_fast_exit, ~0.19 s at 1M). In-process callers (tests) still drop it.
• Head frontier and max counter are cached in the snapshot and carried over the tail (head_memo was ~0.08 s).
• Replica::receive_all integrates every ready op of a batch in one merged undo/redo pass. This also fixed a quadratic replay of multi-replica logs, whose file order is not < order: 65k ops from 54 replicas took 57 s to replay, now 0.23 s (existing since #28, found by the D-030 run). No-snapshot 1M replay also fell from 2.55 s to 2.11 s.

Short-id index not persisted (#32 addendum). It is not needed for the target (status is 0.63-0.67 s at 1M, the index is ~0.07 s of it). Persisting it would add ~56 MB (1M × (32-byte version id + op id)) to the snapshot and need incremental maintenance for every tail, and D-005 output depends on all principal ids. So it is still rebuilt, only by commands that print or resolve ids.

D-030: 100 replicas (partial, stopped at the time cap). Setup as #30: a 5k-file base (~5k ops), 99 clones (own replica ids), each replica, and A, did 20 offline rounds of 50 edits on its own slice (100 × 1000 ops). The build took 861 s. The star sync of replicas 1..99 into A was killed at 900 s after 54 replicas: sync k took 5.4 s (k=10), 10.0 (20), 15.2 (30), 22.0 (40), 27.0 s (50), peak RSS 184-523 MB. At that point A had 65,050 ops over 55 replica ids:

on A (65k ops, 55 replica ids)	wall s	peak RSS MB
status (after checkout)	0.18-0.19	160-164
record 1 edit	0.15-0.27	125-149
log	0.04	145
sync one more replica (r054, r055)	32.6, 26.2	430, 456

• The dense-VV memory question from #28/#30 is not a problem at this scale: status stays at ~160 MB with 55 replica ids, and even sync peaks at ~0.5 GB.
• Sync is the bottleneck, and it is not replay: sample of one star sync shows 86 % in rename(2) in BlobBatch::commit. Each new peer receives all of A's accumulated blobs (~k × 1000 files into one flat blobs/ directory), so star sync is O(N²) in blobs, and the per-file rename into a large flat directory dominates. Recommended follow-up Issue: blob transfer cost in sync (fan-out directories under blobs/, and/or not renaming per blob), then finish the 100-replica run.
• Before the receive_all fix, the first status after the killed sync took 10.4 s (a 54k-op out-of-order tail over the snapshot); after it, 0.07 s.

Issue #32: per-command overhead at 1M ops (2026-09-29)

Branch hub/issue-32 (base develop 28c0a7e), commit 8a4c121. No on-disk format change; goldens unchanged.

Design.

• Conflicts come from the replica. Replica logs each op's fold outcome next to its undo record and keeps the D-014 revive triples from its last view refresh. Replica::conflicts(&store) derives invalid-op, cycle, same-node-move, content, edit-revived and revived from these, using Causality over the CLI store (built only if some node has 2+ applied moves or sets, or there are moves). It does not clone the ops and runs no second fold. If the store keys are not exactly the applied set it returns None and the caller uses evaluate.
• Record/mutate planning uses Replica::state_with_tail(ops), which previews the new head when the new ops are greater than every applied op and no D-014 revive is involved. Otherwise it returns None and evaluate is used. The new frontier is (F(store) ∪ ids(ops)) − deps(ops), so the store is not cloned.
• show of head uses the replayed state, with no down-closure and no evaluate.
• Short-id index: explicit versions (genesis, checkpoints, head) go in a map. The per-op principal versions are in a flat sorted Vec<(VersionId, OpId)>, hashed in parallel threads with no per-entry allocation. short/resolve query both parts as one set, so the output is identical. The principal ids are still all hashed. D-005 short length depends on whether any indexed id shares 8+ hex digits, so skipping them would change output (about 2.3e-4 chance per printed id at 1M). The index is built only by commands that print or resolve an id.
• Repo memoizes the head frontier and max counter, and the FrontierCheck::scan flag, keyed by store size and cleared on every append. read_published reuses the flag through FrontierCheck::with_fast.
• warnings_view builds its replica over only the stored ops that pending ops can query (pending ids and their deps), not a clone of the store.
• Checkpoint: a manual Debug without the memo. Clone resets the memo because a clone may go to another replica's store.

Equivalence.

• hub-core equivalence::replica_conflicts_equal_evaluate (512 cases): Replica::conflicts equals evaluate after each of two delivery batches, and state_with_tail equals evaluate and receive_all. Histories include own-chain-violating and extra-dep ones.
• lazy_checkpoint::split_index_equals_single_map (512 cases) and a unit test with forced 8-64-digit shared prefixes over every principal/map split.
• hub-cli perf32_tests::cli_output_equals_pre32_oracle (512 cases): two copies of one repo run the same edits, record/record -m/commit/publish/checkout/mkdir/touch/mv/rm and foreign concurrent ops. One copy uses the pre-#32 oracle paths (ORACLE). Stdout, stderr, results, logs and working trees of status, status --ignored, log, show, show -l, ls, ls -l and warnings must be equal. Content, edit-revived, same-node-move, name-collision, behind-head and pinned states all occur.
• sync::warnings_view_tests (512 cases) compares the sub-store and the full clone.

Byte identity at 1M. A develop 28c0a7e release binary and the new binary were run on the same 1,000,100-op repo. status, log, ls, warnings and show/show -l (head, a checkpoint, zzzz, 0000) are identical. record -m on two clones gives identical stdout, ops.log, checkpoints.log and status afterwards. At 100k, status, log and ls are identical.

Mutation checks.

• A head memo kept across appends fails the CLI proptest on record -m (wrong label).
• A stale revive set kept after a later receive fails replica_conflicts_equal_evaluate.
• Both were reverted.

Before/after (release, caffeinate, old and new runs interleaved on the same data). 5 runs, median (min-max), wall s. 100k = ops --files 5000 --per 5000 (100,050 ops); 1M = ops --files 10000 --per 10000 (1,000,100 ops, built in 483 s):

command	100k old	100k new	1M old	1M new	1M speedup
ls (open/replay only)	0.245 (0.243-0.265)	0.250 (0.247-0.267)	2.590 (2.516-2.636)	2.550 (2.516-2.580)	1.0x
status clean	0.555 (0.520-0.569)	0.329 (0.322-0.338)	5.894 (5.647-5.971)	3.345 (3.315-3.468)	1.76x
status 1 edit	0.573 (0.537-0.584)	0.339 (0.321-0.349)	5.708 (5.631-5.817)	3.345 (3.299-3.540)	1.71x
record 1 edit	0.548 (0.542-0.576)	0.365 (0.361-0.379)	4.740 (4.666-5.066)	2.904 (2.834-2.917)	1.63x
record -m	0.637 (0.618-0.659)	0.375 (0.368-0.386)	6.160 (5.996-6.469)	3.212 (3.166-3.246)	1.92x
log	0.347 (0.335-0.352)	0.273 (0.271-0.283)	3.619 (3.504-3.823)	2.778 (2.696-2.847)	1.30x
show head	0.480 (0.469-0.489)	0.284 (0.279-0.289)	5.047 (4.918-5.247)	2.761 (2.755-2.851)	1.83x

New floor and replay share. At 1M, replay (ls, 2.55 s) is 76 % of status (3.35 s), 88 % of record 1 edit, 79 % of record -m and 92 % of log. The non-replay remainder is now 0.8 s for status (conflict derivation, index hashing, wd scan, render), 0.35 s for record and 0.23 s for log. Only a snapshot or batch-marker format change (pending a user decision) can remove the replay part.

Issue #31: lazy checkpoint decode, range-based short ids (2026-09-29)

Branch hub/issue-31 (base develop a881afb, develop 51677ad merged in). No on-disk format change.

Design.

• FrontierCheck (hub-core checkpoint). One O(ops x deps) scan per open checks that the store is closed and Lamport. Each checkpoint frontier is then checked for membership only. The greatest missing id gives Missing(id), the same error as the eager downclose. A frontier with 2 or more ids is canonicalized to max≺ with a tree::Causality that is built once and only when needed, so an antichain stays unchanged and a comparable set is absorbed exactly as before (the eager path never rejected it). A store that is not closed or not Lamport falls back to the eager down-closure. Checkpoint::version is computed lazily and memoized.
• VersionIndex::short now does two BTreeMap neighbour lookups instead of hex-formatting every key. The key that shares the longest prefix is a sorted neighbour. resolve is a range scan from the zero-padded prefix. Output is byte-identical: the #13 goldens pass and the old scans are kept as oracles.
• Oracles (test-only): Checkpoint::from_frontier_eager, VersionIndex::short_scan / resolve_scan (hub-core oracle feature), and store::decode_ckpt_eager switched on by store::EAGER (cfg(test)). Equivalence proptests run 512 cases each: core (checkpoints, errors, ids, versions, compare, resolve, short, on closed, non-closed and non-Lamport stores) and CLI (open errors, log, status, show, resolve errors, with random local and published checkpoints including unknown ids and non-antichains). Mutation check: skipping the antichain canonicalization fails both core tests.

Before/after (same machine, release, caffeinate; old = develop a881afb binary on the same data). 5 runs, median (min-max), wall s:

command	100k before	100k after	1M before	1M after	1M speedup
status (clean)	1.008 (1.001-1.108)	0.541 (0.533-0.558)	25.2 (1 run)	5.88 (5.76-5.96)	4.3x
status (1 edit)	1.012 (1.008-1.061)	0.533 (0.526-0.538)	25.6 (1 run)	5.76 (5.72-5.95)	4.4x
record (1 edit)	0.921 (0.894-0.957)	0.527 (0.518-0.536)	23.7 (1 run)	4.82 (4.72-4.90)	4.9x
record -m (3 runs / 1 run)	1.18	0.62	26.1	6.14	4.3x
log (3 runs / 1 run)	3.27	0.31	140.9	3.61	39x

100k = 5k files, 100,050 ops, 18 checkpoints. 1M = 10k files, 1,000,100 ops, 98 checkpoints (the #30 setup). With the new binary the 1M build took 627 s, against 1118 s in #30. Output of log, status and show at 1M is byte-identical between old and new.

Targets. 100k status/record at or below 1 s: met (0.54 s). 1M reduced 10x from 23 s: not met (about 4.4x). The floor below explains why.

log finding. The cause was different: VersionIndex::short scanned and hex-formatted all ~1M keys for every printed id (98 x 1M). It is fixed by the neighbour lookup (item 1 from the Planner).

New floor at 1M (replay). hub ls (open = read + decode ops.log + receive_all replay + checkpoint check, no status work) takes 2.74 s, which is 47 % of status (5.8 s) and 12 % of the old 23 s. Open alone is above the 2.3 s 10x goal, so 10x needs the snapshot Issue. The rest of status at 1M, from a sample profile: a full tree::evaluate for conflicts (about 1 s), Repo::index (about 0.7 s: 1M SHA-256 principal ids plus head()), repeated Repo::head() (Version::from_ops closure check plus frontier), FrontierCheck::new twice (open plus read_published), and a store clone in warnings_view. At 100k, ls is 0.23 s (43 % of status).

Issue #30: 1M ops, replica run, 100k re-measure (2026-09-29)

develop a881afb plus hub/issue-30 fd5e29b (only message text and a test-seed env changed; no performance change). Same machine, release, caffeinate -i, per-command timeout 900 s. No optimization in this Issue.

1M ops. hub-bench ops --files 10000 --per 10000 --targets 100000,1000000. The Issue suggested 1k files, but with at most one op per file per record that would need about 1000 record rounds. 10k files keeps the build to 100 rounds. The build took 1118 s (18.6 min, under the 20 min cap), and the slowest round took 25.9 s. .hub was 233 MB with 98 checkpoints. Wall s / peak RSS MB:

command	100,100 ops	1,000,101 ops	D-026 target	pass at 1M
status (clean)	0.83 / 365	23.29 / 3172	<= 1 s	FAIL (23x)
status (1 edit)	0.82 / 366	23.60 / 3287	<= 1 s	FAIL
record (1 edit)	0.77 / 389	22.32 / 3050	<= 1 s	FAIL
log	1.54 / 236	138.19 / 2390	-	slow
sync into empty repo	6.26 / 599	52.47 / 5213	<= 60 s (1-week offline)	ok (barely)
checkout into empty dir	3.30 / 338	9.12 / 2882	-	-

Open/replay is not a separate command. status, record and log each open and replay the log, so their times include it. The profile below splits open from the rest.

Profile, status at 1M ops (macOS sample, 20 s, 14,975 samples in hub_cli::run):

• 94.7 % (14,184) is in Repo::open_dir_with → store::decode_ckpt → Checkpoint::from_frontier. That splits into Version::downclose (7,557) and Version::frontier (6,389). Each of the 98 checkpoint records is rebuilt with a full down-closure over the op log: O(checkpoints x ops), about 10^8 at 1M ops. The rest is BTreeMap insert, memmove and malloc under those two calls.
• 3.6 % (535) is the actual op replay (Replica::receive_all, buffer, refresh_view).
• Everything else, including evaluate and the working-dir scan, is under 1 %.

Ranked causes (1M-op status/record fail):

1. Checkpoint decode on open is O(checkpoints x ops) (store::decode_ckpt → Checkpoint::from_frontier → downclose + frontier; about 95 % of samples). This cause was not on the Issue's list. It also explains why 100k ops with 5k files (18 checkpoints) is slower (1.18 s) than 100k ops with 10k files (8 checkpoints, 0.83 s). Candidate fix, for a later Issue: trust the stored frontier, validate lazily or once with a single linear pass, or persist checkpoint versions.
2. Full replay without snapshots (receive_all about 4 % of samples). This is small today, but it becomes the floor once cause 1 is fixed. It is bottleneck 1(c).
3. Memory. Peak RSS is about 3 GB for status and 5.2 GB for sync at 1M ops (about 3 KB per op), which is roughly linear from 100k (365 MB). The profile does not separate the dense VV (Causality) memory. The replica run below was meant to test that and did not finish.

• The Revived pass O(D x M) did not show in the profile (not a cause here: no concurrent delete/move).

100k ops x 100 replicas: partial, stopped by user decision (time). A script built a 5k-file base (about 10k ops) plus 99 clones. Each replica, and A, made 20 offline record rounds of 50 edits on its own 50-file slice, for about 1000 ops per replica and about 110k ops in total over 100 replica ids. Building the replicas took 714 s. The star sync of the 99 replicas into A was then stopped, so no status, record or log numbers at 100 replicas were taken. The Causality O(ops x replicas) memory question (#28 open point) is still open.

100k-op status/record re-measure (item 4). hub-bench ops --files 5000 --per 5000 --targets 100000 was run 5 times (fresh repo each run, 100,050 ops, 18 checkpoints). Wall s:

command	median	min	max	runs
status (clean)	1.178	1.152	1.185	1.178, 1.152, 1.181, 1.177, 1.185
status (1 edit)	1.187	1.127	1.217	1.127, 1.144, 1.208, 1.187, 1.217
record (1 edit)	1.065	1.031	1.136	1.031, 1.049, 1.136, 1.085, 1.065

The spread is small (at most 8 %), so the 1.1-1.2 s reading from #29 is real, not noise: at 100k ops with 5k files, status and record are just over the D-026 1 s target. Peak RSS was 321-369 MB. The gap from #28's 0.83 s is explained by checkpoint count (cause 1 above), not by #29.

After #29: group-commit durability (2026-09-29)

One barrier per batch instead of one F_FULLFSYNC per blob, file and op; same durability contract (Repository-Semantics Appendix A.1). No on-disk format change. Same machine, release, caffeinate -i, per-command timeout 600 s. "Before" is develop fd8138f (#28 merged) rerun in the same session. "After" is hub/issue-29 da03f14. Wall time in seconds.

Case	Before	After	Speedup
1k files: first record	22.81	0.40	57x
1k files: sync into empty repo	23.37	0.40	58x
1k files: checkout into empty dir	6.32	0.35	18x
10k files: first record	222.68	3.71	60x
10k files: sync into empty repo	228.05	3.54	64x
10k files: checkout into empty dir	64.61	3.07	21x
100k ops: sync into empty repo	595.65 (just under the 600 s cap)	5.21	114x
100k ops: checkout into empty dir	31.56	1.95	16x
100k ops: bench build (5k files, then records)	644	27	24x

• D-026 "sync after 1 week offline <= 1 min": 100k-op sync into an empty repo is 5.2 s, so it now meets the target at this size. 1M ops was not run.
• status/record at 100k ops read 0.9 s before and 1.1-1.2 s after. #29 does not touch those paths (no fsync on status), so this is run-to-run noise or replay cost, and it sits at the edge of the 1 s target. Bottleneck #1(c) (snapshot) is the remaining lever.
• Peak RSS is unchanged except 100k-op sync (604 → 580 MB).
• Linux has not been measured: per-file fsync there already includes a cache flush, so the gain will be smaller.

After #28: O(n^2) evaluate/replay removed (2026-09-28)

Algorithm change only; observable semantics unchanged (equivalence proptest against the pre-#28 oracle, 1024 cases; golden vectors unchanged) and no on-disk format change. Same machine, release, caffeinate -i, per-command timeout 600 s. "Before" is develop 19d198f rerun in the same session (RSS reads higher than in the #27 table below; same binary, so the difference is run-to-run and memory-pressure noise).

What changed (hub-core):

• tree::Causality: per-op version vectors (pointwise max over deps, exact for any op set). A VV answer "a is before b" is trusted only for replicas whose own chain (§2.6 #7) holds, checked while building; for a replica that violates it, an exact pruned walk of the deps DAG is used. Replaces the materialized causal_past in evaluate and in the D-014 view refresh.
• evaluate no longer clones the whole state before every Move/Delete: the pre-delete parent is read during the fold, and the Revived ancestry check runs in a second fold only when a concurrent delete/move pair exists.
• Replica::receive_all: the view is refreshed once per batch (open replay, record, sync persist), and pending ops use a ready index instead of a scan per op (also in SyncReplica).

Wall s / peak RSS MB:

command	1k before	1k after	3k before	3k after	10k before	10k after	10k speedup / RSS
first record	21.8 / 132	24.0 / 12	66.5 / 1137	70.1 / 35	250.9 / 5942	215.8 / 106	fsync-bound
status (clean)	0.14 / 135	0.04 / 12	1.30 / 1144	0.11 / 30	19.4 / 8281	0.26 / 98	75x / 84x
status (1 edit)	0.14 / 135	0.03 / 11	1.30 / 1145	0.06 / 30	19.0 / 10359	0.20 / 96	97x / 108x
record (1 edit)	0.18 / 135	0.07 / 13	1.35 / 1147	0.12 / 35	18.8 / 10772	0.29 / 116	66x / 93x
log	0.06 / 8	0.02 / 8	0.42 / 18	0.03 / 19	6.2 / 60	0.11 / 62	56x
sync into empty repo	19.2 / 20	22.6 / 19	72.9 / 51	68.9 / 51	252.4 / 165	217.5 / 141	fsync-bound
checkout into empty dir	5.7 / 135	6.3 / 14	19.2 / 1145	18.3 / 35	88.0 / 8851	72.5 / 103	fsync-bound, RSS 86x

Scaling of status (clean) after #28: 0.04 → 0.11 → 0.26 s for 1k → 3k → 10k files (x3 and x3.3 files give x2.8 and x2.4 time): about linear. Before it was x9.3 and x15 (quadratic).

Ops axis (5k files, 256 B):

command	10,050 ops before	10,050 ops after	100,051 ops after
status (clean)	3.98 / 3152	0.13 / 45	0.83 / 354
status (1 edit)	3.99 / 3152	0.08 / 41	0.84 / 353
record (1 edit)	4.09 / 3156	0.16 / 53	0.72 / 323
log	1.32 / 28	0.06 / 29	2.76 / 248
sync into empty repo	120.0 / 71	122.2 / 69	TIMEOUT > 600 s (fsync-bound)
checkout into empty dir	31.5 / 3153	32.3 / 46	31.9 / 338 (of the partial clone; not meaningful)

The 100k-op log was built in 509 s (18 record rounds of 5k edits, slowest round 29 s; before #28 one such round took ~4-6 min at ~60k ops). 1M ops not run.

Now dominant: bottleneck #2 (one F_FULLFSYNC per blob and per op). First record, sync and checkout did not change (they were fsync-bound already; only their RSS dropped); sync of 100k ops times out. Not fixed here (separate Issue). Remaining CPU note: evaluate's Revived pass still pairs each winning delete with each applied move (O(deletes x moves) cheap VV tests), and log at 100k ops (2.8 s) still replays; snapshots (1c) are separate.

Machine

MacBook Pro Mac15,11, Apple M3 Max, 14 cores, 36 GB RAM, internal SSD (Apple Fabric), macOS 15.6.1, APFS, rustc 1.87.0, release build. Runs used caffeinate -i. Other factory sessions were active, so expect some I/O contention. The status-at-10k profile ran at the same time as a sync (under a shared lock).

Targets (Planner default)

100k files, 10 GB, single files up to 1 GB; 1M ops; 100 replicas and 20 concurrent editors; status and record <= 1 s; sync after a week offline <= 1 min.

Results (wall time s / peak RSS MB)

command	1k files (~2k ops)	10k files (~20k ops)	100k files	target	pass
init	0.34 / 2	0.06 / 2	0.12 / 2	-	ok
first record	22.6 / 132	253.7 / 3825	TIMEOUT >300 s	-	FAIL
status (clean)	0.16 / 134	20.1 / 4187	not run	<= 1 s	1k ok, 10k FAIL
status (1 edit)	0.14 / 134	19.8 / 6048	not run	<= 1 s	10k FAIL
record (1 edit)	0.19 / 135	20.1 / 6280	not run	<= 1 s	10k FAIL
log	0.06 / 7	6.4 / 57	not run	-	10k slow
sync into empty repo	22.4 / 17	246.0 / 162	not run	-	FAIL
checkout into empty dir	6.5 / 135	80.6 / 3796	not run	-	FAIL

Ops axis (5k files, 256 B each, edits that dedup the blobs so only the op log grows):

command	10,050 ops	~55-65k ops (~ partial)	100k ops	1M ops
status (clean)	5.5 / 1868	-	not reached	not run
status (1 edit)	4.8 / 2816	-
record (1 edit)	4.7 / 3156	~ one 5k-edit record round took about 4-6 min (from ps etime; partial observation)	not reached: axis stopped	extrapolated unusable
log	1.6 / 29
sync into empty repo	110.1 / 65
checkout into empty dir	41.6 / 1941

Sync of a delta (A and B share a 1k-file base, B makes K ops offline):

K	sync	checkout after	target
10	0.39 / 16	0.27 / 136	ok
1,000	5.96 / 22	7.46 / 293	ok (<= 60 s)
100,000	not run (extrapolated: >= 100k fsyncs, about 10 min, plus O(n^2) replay)		FAIL (extrapolated)

Blobs (reduced to 150 MB, see scope cuts):

shape	record	status	sync into empty	checkout
1 x 150 MB	0.92 / 152	0.37 / 152	1.87 / 452	0.50 / 152
150 x 1 MB	3.66 / 18	0.02 / 6	5.33 / 447	1.55 / 157

Whole blobs are held in memory. RSS is about 1x the blob size for record and checkout, and about 3x for sync. Extrapolating, a single 1 GB file would need about 1 GB of RSS for record and about 3 GB for sync, with a wall time of about 6-12 s.

Watch idle (--poll 100, 30 s, 1k files): 9.9 % of one core, 9.9 ms CPU per poll, RSS 29 MB. The poll walk is O(files). 10k files would be roughly 100 % of a core (extrapolated; not run).

Scenario runner (100-file base, 2 offline record rounds per replica, then sync passes until heads agree):

run	passes	syncs	total sync	slowest sync	converged (head + tree)
R=2 ring	1	2	0.75 s	0.57 s	yes
R=2 star	1	1	0.66 s	0.59 s	yes
R=10 ring	2	20	30.9 s	2.71 s	yes (with conflicts)
R=10 star	2	18	28.4 s	2.64 s	yes (with conflicts)
R=50	not run				extrapolated from the 10k-file results

Every run that was executed converged: the head ids matched and the checked-out tree hashes were identical.

Scope cuts (user decision on 2026-09-28, relayed by the Mentor)

Once the bottlenecks were clear, the runs were cut to fit the time budget. The ops axis stopped during the 100k build (the run was killed while building) and 1M was not run. 100k files stopped at the record TIMEOUT. The delta runs used a 1k-file base and K <= 1000. Blobs were reduced to 150 MB instead of 1 GB. Watch was sampled for 30 s at 1k files only. Scenarios ran R = 2 and 10 only.

Profiles (macOS sample, release binary)

• first record, 1k files: 70 % of samples are in fcntl (F_FULLFSYNC from File::sync_all) inside Blobs::put, which does a temp-file fsync plus a directory fsync per blob. Most of the rest is open and rename. sync and checkout show the same pattern: Blobs::put, sync::persist and wd::apply spend 60-70 % of their time in fcntl. There is also one append_record fsync per op.
• status, 10k files: Replica::receive self time is about 60 % and causal_past about 30 %, with BTree inserts and malloc making up the rest (profile taken while a sync held the shared lock).
• status, 3k files (#27 Review rerun): causal_past is about 45-50 % of samples and is reached from tree::evaluate (the conflict derivation that status, record and sync run on every call), not only from refresh_view; Replica::receive is about 15 % (per-op refresh_view, whose fast-path check is itself an O(n) log scan).

Bottlenecks, ranked

1. Evaluate and open-time replay are O(n^2) in time and memory (fixed by #28, see above). The primary O(n^2) memory source is tree::evaluate's causal_past (the conflict derivation run by status, record and sync): it materializes a transitive predecessor set per op, Σ i = n^2/2 entries on a linear history (4-6 GB at 20k ops). Secondly, open replays the whole op log op by op (Replica::receive), calling refresh_view after every op; its fast-path check scans the whole log (O(n) per op), and it rebuilds causal_past too when some SetContent targets a non-live node. The pending scan was cheap during in-order replay. This is why status and record took 20 s at 10k files, and why 100k files and 1M ops were unreachable. Fixes: (a) batch receive so that it integrates everything and refreshes the view once per command; (b) replace the materialized causal_past with version vectors or Lamport and replica frontiers for concurrency tests, or compute it only for the ops that D-014 revive needs; (c) persist a snapshot or checkpointed projected state plus an index so that open does not replay history. Estimated impact: (a)+(b) take status at 10k from 20 s to under 1 s and memory from GB to tens of MB; (c) makes open O(recent ops).
2. One durable F_FULLFSYNC per blob and per op (fixed by #29, see above) (blobs.rs put: file plus directory fsync; store.rs append_record; wd.rs apply). The cost is about 10 ms per file, so first record, sync and checkout run at about 45 files/s, which is 250 s for 10k files and an estimated 40 min for 100k. Fix: group commit. Write all blobs and ops of a command, then fsync once (one blob-dir fsync and one log fsync per command), or use packfiles. The crash-safety invariants stay intact if the log fsync still comes after the blob fsyncs. Estimated impact: 10-100x on record, sync and checkout of many files.
3. The watch poll walks the whole tree (watch.rs fingerprint), which costs 10 % of a core at 1k files and grows linearly. Fixes: FSEvents or inotify (the notify crate, already weighed in the #25 module doc) with the poll kept as a fallback, and/or a longer default poll. Estimated impact: idle CPU close to 0.
4. Whole-blob in-memory handling in sync (RSS about 3x the blob size). Fix: stream blobs (copy file to file with a hashing reader). This matters for the 1 GB single-file target.
5. log replays too (6.4 s at 10k files). It only needs checkpoints, and would be fixed together with 1(c).

Rerun

See crates/hub-bench/README.md on the branch: cargo build --release, then for example caffeinate -i timeout -s KILL 1500 target/release/hub-bench files --root <dir> --n 1000,10000 --timeout 600. The cases are files, ops, delta, blobs, watch and scenario. The bench is a separate binary with test = false, so it never runs under cargo test.

#46 divergence digest (2026-10-14)

Release build, macOS (14 cores), develop c8be24d (old) vs hub/issue-46 (new), the same repository for both (no format change), runs interleaved old/new, median of 5. The repo has 10k files, edited in rounds; the ops are SetContent. Sync is file:// (both sides in one process).

ops	status	record (1 edit)	sync into empty	no-op sync
110k	0.372 → 0.367 s (-1.3 %)	0.339 → 0.344 s (+1.5 %)	18.9 → 19.3 s (+2.1 %)	1.60 → 1.63 s (+1.9 %)
1.02M	0.990 → 1.039 s (+4.9 %, code path unchanged: noise)	0.829 → 0.823 s (-0.8 %)	36.0 → 37.0 s (+2.8 %)	7.59 → 7.69 s (+1.3 %)

The first version (single-threaded hashing plus a per-op hash memo and a set of the delta's ids) gave +11.4 % (sync into empty) and +9.0 % (no-op sync) at 1M. That is over D-026's 10 % bar, so it was replaced by parallel hashing without a memo and a merge walk for K.

#46 repair (Review R2), 2026-10-14

Same setup (develop c8be24d vs the repaired hub/issue-46, interleaved, medians of 5). The machine had a load average of about 9 from other sessions.

• LtHash16 on the wire (2 KiB BLAKE3 XOF per op, about 2.1 µs/op/core): no-op sync at 1M ops was +11.8 % and +12.3 % in two runs, over D-026's 10 %. Rejected for the full-replica path.
• SHA-256 sequence digest on the wire (final):

ops	status	record (1 edit)	sync into empty	no-op sync
110k	0.331 → 0.325 s (-1.7 %)	0.331 → 0.334 s (+1.0 %)	17.06 → 17.20 s (+0.8 %)	1.559 → 1.658 s (+6.4 %)
1.02M	1.082 → 1.047 s (-3.2 %)	0.814 → 0.788 s (-3.2 %)	38.61 → 38.64 s (+0.1 %)	8.650 → 8.279 s (-4.3 %)

#53 text CRDT core: Text-CRDT §7.11 B1 and B3 (2026-10-15)

Release build, Apple M3 Max, branch hub/issue-53 (commit b8ace57), in process on hub-core (caffeinate -i cargo bench -p barocss-hub-core --bench text). This is a core bench, not a hub-bench case, because hub-bench never links product code. Memory is the peak of a counting global allocator during the replay (heap bytes, not RSS).

bench	workload	result	target
B1 line level	deterministic synthetic equivalent of the automerge-perf trace (the trace itself is not vendored): 260 000 keystrokes with cursor locality on a ~140 KB document, one TextEdit per keystroke (replace the touched line / new line / delete line), replayed into a fresh doc	0.107 s, 70.4 MB, 254 724 runs, text matches	≤ 0.5 s, ≤ 100 MB: met
B1 char level (reference)	the same keystroke model, one op per character	0.093 s, 66.5 MB, 234 022 runs	none
B3	20 editors × 500 line edits on one 10 000-line file, 20 sync rounds; 3 random causal delivery orders into fresh docs	converged in all 3 orders; integrate per op p50 0.0007 ms, p99 0.0022 ms, max 0.196 ms	convergence, p99 ≤ 1 ms: met
B3, full evaluation	evaluate_with over all 10 002 ops (validity, content, conflicts, notices)	0.23 s; 10 000/10 000 valid; 13 notices	none

The first TextDoc layout (an OpId and three Vecs per fragment, per-op hash maps) replayed B1 in 0.166 s but used 179.4 MB, which missed the target. The compacted layout uses interned op indices, a byte arena, linked sibling lists and inline one-deleter sets.

#54 TextEdit storage and wire: Text-CRDT §7.11 B2, B5, B1 rerun (2026-10-15)

Release build, Apple M3 Max (36 GB), branch hub/issue-54, in process on hub-core (caffeinate -i cargo bench -p barocss-hub-core --bench text -- b2). The workload goes through the incremental text-aware Replica and the snapshot v2 payload (Repository-Semantics A.2.9). Memory is the counting allocator's peak (heap bytes).

bench	workload	result	target
B2 open with snapshot	10 000 files of 2–200 KiB (uniform; 978 MiB of seeds in blobs/), 1 000 000 TextEdits (20 editors, each an own causal chain; one line inserted per edit at a random seed offset, every 5th edit also deletes the file's previous insert), 1 020 000 ops: read the snapshot file, decode the replica, check the 10 000 referenced seed blobs exist (stat)	read 0.01 s + decode 0.61 s + seed checks 0.04 s = 0.66 s, peak 1.8 GB	≤ 1 s: met (core payload; see note)
B2 record step	one more TextEdit received by the reopened replica (loads that file's seed from blobs, validity, integration, re-derive of its node; includes building Causality over 1M ops)	0.41 s	≤ 1 s: met
B2 open without snapshot	full replay of the 1.02M ops into a fresh replica (all seeds supplied)	9.9 s	none (the snapshot is what makes open fit)
B5 snapshot size	the same replica; seeds referenced by blob id	100.3 MiB; live inserted bytes 13.4 MiB, 2 008 849 runs: 1.2 × (13.4 MiB + 40 B × runs) = 108.0 MiB, so size is 0.93 of the bound (no fixed overhead needed); encode 0.24 s	≤ 1.2 × (live inserted + 40 B × runs) + fixed: met
B5 first version (superseded)	seeds copied into the snapshot	1078 MiB (978 MiB seeds), open 0.81 s	missed the size target (Planner)
lazy loads	loading every one of the 10 000 lazy docs afterwards (reads 978 MiB of seeds)	2.3 s in total, spread over first uses	none
B1 line level (rerun)	synthetic automerge-perf equivalent (unchanged from #53)	0.077 s, 70.4 MB, text matches	≤ 0.5 s, ≤ 100 MB: met
B1 char level (rerun)	same, one op per character	0.061 s, 66.5 MB	none
B3 (rerun)	unchanged from #53	converged; p99 0.0018 ms	met

• B1 on the real automerge-perf trace: pending user approval of the download. The Planner relayed the user's approval (Issue #54 comment), but the Compute session's own download permission was not granted, so the trace was not fetched. B1 is met on the synthetic equivalent only.
• Note on B2: the 0.66 s covers the snapshot file read (100.3 MiB), the payload decode and the 10 000 seed stats. The CLI open also checks the 64-bit checksum of the file and binds it to ops.log (A.2.3). That pass and a full CLI status/record at B2 scale (≤ 1 s, D-026) are not measured here. Per Review they are a T3 acceptance item (#56), since T3 brings text record/checkout to the CLI.
• Fixes found by this bench: the first v2 decode took 1.84 s. Per-op Vecs for inserted bytes plus a binary search of the log per doc op cost about one cache miss per step. Now there is one arena allocation per doc and one index over the log's text ops, and the doc's internal maps use a fast hasher. Ops that break §2.6 #7 (non-chained authors) make Causality::precedes walk the DAG, which made full replay 19 s. That was a bench artifact, since real authors chain their own ops, and it is fixed in the generator.
• The snapshot references each epoch's seed by blob id (the 7.11 alternative, made the target by the Planner): open no longer reads or copies 1 GB, and each seed is read on first use.