Measures L0 Engine

Measures: the L0 engine

The first measurements of ehdb-l0, the engine on the production path. Numbers as of 2026-10-08; re-measure rather than quoting them.

cargo bench -p ehdb-l0 --bench l0_engine
cargo test  -p ehdb-l0 --test invariant_measures -- --nocapture

Methodology, hardware and the full table live in the repo at docs/measures/l0-benchmarks.md. Landed by noetl/ehdb#381.

Why these are the first

crate	.rs files	bench files
ehdb-l0 — the production engine	88	0
ehdb-feed	36	0
ehdb-reference — the reference model	22	2
ehdb-service / -transaction / -catalog / -storage	3/3/2/2	1 each

82% of the workspace had no benchmark, and the production engine had none while the reference implementation had two. Design: Performance and load testing measures ehdb-reference (cargo bench -p ehdb-reference), so every number on that page is about the model. These are about the engine.

⭐ And the two agree where they overlap. That page measured durable append as "fsync-bound at ~3.9 ms → ~256 appends/s"; the production engine measures 3.749 ms → 267 appends/s — within 4%, from a different crate and a different harness. Two independent instruments converging on one figure is the strongest form this evidence takes.

Result 1 — the fsync is ~95% of posture-A write cost

path	records/s	µs/record
posture A (EveryAppend)	267	3749
group commit (CallerDriven), batch 8	1 010	990
group commit (CallerDriven), batch 80	5 450	184

20.4x. This is new — the reference page has no group-commit figure.

⚠ Group commit requires engine.set_flush_policy(FlushPolicy::CallerDriven). FeedWriter does; a bare L0Engine caller must too, and without it batching costs more (the batch sync becomes a second fsync on top of the per-append ones).

Result 2 — a read steps 15.6x per record at seal_max_records = 1024

records on one index key	time	ns/record
10	711 ns	71.2
100	5.52 µs	55.2
900	49.1 µs	54.6
1 100	933.8 µs	848.9
2 200	1.888 ms	858.3
4 400	3.760 ms	854.6

A 22% increase in length costs 19x. Below DEFAULT_SEAL_MAX_RECORDS a key's records sit in the in-memory active part; above it, in a sealed part on disk. Per-record cost steps 15.6x and is then flat — a one-time step, not quadratic.

⚠⚠ An execution chain over 1024 events costs ~13x what extrapolating from the sub-1024 figures predicts. See Design: Execution chain store — a long-running execution is exactly the case that crosses the boundary. Tracked as noetl/ai-meta#453, where whether the step is acceptable is an open judgement call, not an assumed defect.

Lifecycle — and the manifest quadratic IS fixed

The highest-risk surface. On 2026-09-01 the manifest path reached 6 770 snapshots / 19.4 GB behind 71.8 MB of real data, filled the prod volume and made every append fail. manifest_retain (noetl/ehdb#344) was the fix. Measured as a growth exponent (bytes ~ appends^e; e=1 linear, e=2 quadratic), because a raw ratio cannot be compared across span sizes:

	bytes	files
retain=32 (default)	1.23	0.08
retain=0 (pre-fix behaviour)	1.94	1.01

⭐ The control is a supported configuration. manifest_retain = 0 disables pruning — it is the pre-fix behaviour by its own doc comment — so the planted defect is a config the engine still accepts, not a mutation. The pre-fix setting still measures 1.94, which is what makes the verdict on the fix trustworthy.

File count is held flat at ~39 across 4x the appends (35 → 39 → 39) while unbounded climbs 79 → 159 → 320.

⚠ Residual: bounded bytes still grow faster than linear (e = 1.23). Retention bounds how many snapshots exist, not how big one is. Not pathological, not free, and calling it "linear" would be wrong.

⚠ The lifecycle calls are caller-owned. Driving seal_aged_parts / run_pending_merges / reclaim_orphans each seal over 1 024 appends performed 15 merges; one call at the end performed 8. A measure that never drove them would report 0 and read as "no merges needed" rather than "nobody asked".

Tail latency

Criterion gives mean/median and no percentiles. 400 samples, nearest-rank, n printed on every line — a percentile over an unstated sample count is a max wearing a percentile's name.

path	p50	p90	p99	max
append, posture A	4 014 µs	4 815 µs	7 550 µs	11 706 µs
append, group-committed	21.4 µs	50.4 µs	75.2 µs	121 µs
read, len 900	66.0 µs	110 µs	248 µs	928 µs
read, len 1 100	1 910 µs	2 013 µs	2 364 µs	8 409 µs

• the fsync tail is real: p99 is 1.9x p50, max 2.9x — "3.7 ms per append" occasionally takes 11.7 ms;
• group commit is 187x cheaper at p50, not the 20.4x the throughput figure shows (that one includes the batch sync; the append itself skips the fsync);
• the seal boundary is 29x at p50, against 19x in the mean.

No latency threshold is asserted — that would be flaky on shared CI hardware, and a flaky measure gets deleted.

The standard: "complete and measured"

docs/measures/complete-and-measured.md scores EHDB against checkable acceptance criteria — A invariants enforced, B costs known with a resolution control on each instrument, C health observable so a 0 means healthy not inert, D regressions loud.

A nearly met · B met except memory/concurrency/feed · C barely started · D met except CI not running the benches.

⚠⚠ C is the honest headline: metrics.rs exports 7 public functions for an 88-file engine, so most of this engine's health cannot be read from its metrics — noetl/ai-meta#454. And two A criteria are unmet: fold determinism has no engine-side enforcement point (47 mentions, 0 on an assert line), and single-root is not an EHDB invariant at all — zero mentions in this repo; it lives in noetl/server.

Invariant measures

Quantified, with a planted defect on each so a clean result is evidence rather than an absence of it. See Consistency invariants for what the invariants are; this is how they are measured.

measure	healthy	planted control
ordering	0 descending pairs over 500 records, compared == population	a reversed read must flag every adjacent pair; a single swap exactly one
fold determinism	1 distinct digest over 5 reads of 300 records	the fold must change on one swap and on one dropped record, or determinism over it is vacuous
reopen parity	set equality over 400 records	removing one element must fail the comparison

Each prints population=N, because zero violations over zero records is what a broken measure looks like.

⚠ Three of three append instruments were wrong before they were right

Recorded because it is the method, not an anecdote:

1. iter_batched rebuilt the engine per batch → setup dominated at 8.3 ms/append. Caught by its own control failing: a 10x payload came out faster, CIs overlapping.
2. iter_custom fixed it; 267/s still looked implausible — until the engine's own docs confirmed fsync-per-append, posture A at "~4 ms". The number was right.
3. Group commit read flat because the posture was left at EveryAppend. The instrument was wrong, not the engine — "EHDB's batching doesn't amortise" was nearly published.

A timing measure written as a cargo test was removed: debug builds put a ~100 µs fixed floor under it, so a 10x length step cost 1.4x and failed its control. Timing belongs in the release benchmark.

Memory — the engine is O(parts), not O(records)

crates/ehdb-l0/tests/memory_measures.rs, via a counting #[global_allocator].

records	parts	live bytes	B/record
500	1	17,319	34.6
2,000	7	29,484	14.7
8,000	31	73,482	9.2

Growth exponent over record count 0.52 — sublinear. Two-point fit over parts:

~16.6 KB fixed + ~1.8 KB per part. 1 M records at seal_max_records=1024 ≈ 976 parts ≈ 1.7 MiB held.

What this means for operating EHDB: footprint is predicted by part count, so manifest_parts (the new gauge) is the memory signal, and merge is the memory control. A rising manifest_parts with a flat merges counter is the shape that filled the prod volume in ehdb#344.

Sealing is confirmed to be the mechanism that releases, not assumed: the same 8,000 records with sealing disabled hold 331,604 B vs 73,482 B (4.5x).

⚠ The file holds exactly one #[test] and must keep holding one. The allocator counter is process-global and cargo test runs test fns in parallel threads, so a second measuring test would corrupt both.

Concurrency — and a caller contract that is easy to break

crates/ehdb-l0/tests/concurrency_measures.rs. Before it, no test in ehdb-l0 had ever spawned a thread.

append_record(&mut self) — a single L0Engine is a single writer by construction. Behind one Mutex, throughput falls from 294 to 263 appends/s going 1 → 8 threads: threads buy nothing and cost lock handoff. The absolute figure is dominated by the substrate's per-append durability, not the mutex.

⚠⚠ For callers: mint the sort key UNDER the append lock

pattern	out_of_order_appends	lost	duplicated
sequencer minted under the lock	0 of 1,600	0	0
sequencer minted before the lock	547 of 1,600 (34.2%)	0	0
per-thread sequence ranges	473 of 1,200 (39.4%)	0	0

An AtomicU64 plus a Mutex is not sufficient:

let s = seq.fetch_add(1);     // A gets 5, B gets 6
let mut g = engine.lock();    // B wins the race
g.append_record(..s..);       // 6 is appended before 5

Nothing is lost and nothing is duplicated — ids stay unique and dense, only their order is wrong. This is the engine-level reproduction of the mechanism that rolled back the v0.4.4 chain-store ramp (see this wiki's Sessions-Log, 2026-09-30, and noetl/ai-meta#362): a snowflake id minted before insert means commit order is not id order. It reproduces here in 20 ms.

Replay stays strictly ascending in every row — the engine sorts on read — so the damage is to the tail canary, not to read order. out_of_order_appends is the only signal that sees it.

Readers against a writer

3 readers vs 1 writer over 400 appends: 142,945 reads, 142,935 strictly mid-write, 0 invariant violations. The test fails at zero mid-write reads, because a clean result with no interleaving means the writer finished first and proves nothing.

Multi-shard

8 shards, 4 writers, keys shared so every shard is written by every thread: the union over shards is set-equal to what was appended, and each record re-hashes to the shard it was read from.

The feed drain path

crates/ehdb-feed/benches/drain.rs — the first benchmarks in ehdb-feed (36 files, 0 before). Draining a 4,000-record backlog:

batch limit	total	per-record
1	1.353 s	2.95 K/s
256	6.47 ms	617 K/s
1024	2.55 ms	1.56 M/s
4000 (one poll)	2.55 ms	1.57 M/s

530x between batch 1 and batch 1024.

⚠ The mechanism is NOT attributed. The obvious reading — cost = poll count × backlog — is refuted by its own control: 4 polls of 1,024 and 1 poll of 4,000 agree to 0.1%. A second probe at limit 1 shows per-poll cost rising 32x for a 10x backlog, so it is not a fixed overhead either. Tracked as ehdb#389; the docs state it as unattributed rather than guessing.

⚠ default_batch_limit() = 2,000 makes a larger drain quadratic (exponent 2.06). The cap exists because unbounded is what caused noetl/ai-meta#298. Both are real and the tradeoff had never been measured.

⚠ poll_assign without acking is 2.86x slower than with. inflight is a BTreeMap keyed by sort key, not deadline, so the expiry scan walks every entry on every poll — O(k) per delivery, O(n²) for a lazily-acking consumer. Tracked as ehdb#388.

Queue depth and divergence are now scrapable

Four state gauges on L0Metrics (current depth, as against 27 cumulative counters), all computed from in-RAM state with no substrate I/O:

gauge	meaning
ehdb_l0_manifest_parts	live parts — the memory predictor above
ehdb_l0_parts_local_only	sealed, no durable copy — upload backlog
ehdb_l0_parts_under_replicated	some but not enough copies
ehdb_l0_dedupe_window_records	records in the idempotency window

⭐ parts_under_replicated closes a hole in this engine's durability reporting. The uploader assigns p.replicas = locations — the successful writes only — so a part that landed 1 of 2 copies is is_durable() == true, on_upload_done fires, and the age-based durability window reports it done. replica_writes is cumulative and climbs while the deficit stands. Proven in tests/state_gauges.rs with a write-refusing substrate: 7 parts, each holding 1 of 2 copies, each reported durable.

⚠ Refreshed on manifest mutations (open, seal, merge, reclaim), not per append. A per-append manifest walk would be O(parts) per append — the very quadratic manifest_parts exists to detect. L0Engine::refresh_state_gauges() is public so a scrape handler can force one. Pinned unconditionally at open, including all-zero.

And on the delivery side, ehdb_feed_shard_inflight / ehdb_feed_total_inflight now expose ShardConsumerGroup::inflight_len, which existed with no render_* emitting it. It is not derivable from lag, which counts undelivered plus unacked:

lag	inflight	state
10	0	stalled — consumers absent or not polling
10	4	busy — saturated, making progress

Measured: identical lag of 10 in both.

⚠ The existing ehdb_feed_shard_lag / _committed / _total_lag lines are asserted byte-identical — KEDA's metrics-api scaler matches a prefix of the whole name{labels} token with no label selector, so changing their bytes breaks every ScaledObject pointing at them.

The benchmarks run in CI

.github/workflows/ci.yml gained a bench job. CI previously ran cargo bench --workspace --no-run: it compiled every bench and executed none.

Running them is most of the value, because each case asserts inside its timed region that it got the records it claims — which caught ChangeFeed::poll returning 2,000 of a 10,000 backlog. A short read and a fast drain are otherwise the same number.

ci/bench_ratios.py gates shape, not time:

• denominator first — expected=13 parsed=13, failing on a missing benchmark, because a parser that finds nothing asserts nothing and exits 0;
• monotonicity — a larger batch must never be slower (25% tolerance), compared within one run on one runner so machine speed cancels;
• reported, never gated — the 530x and friends. Gating them would fail CI the day somebody improves the drain path.

⭐ First run, local vs runner: batch ordering 530.5x vs 528.3x (machine-independent); per-poll cost 32.0x vs 16.4x (factor of two). A "per-poll ≥ 25x" threshold would have failed on the runner, not the code — which is why only the ordering is gated.

⚠ No baseline file is committed. Checked-in absolute times from one machine are a representation nothing keeps true. Per-run estimates upload as an artifact.

Vectors (P6) — and the compaction fix that followed

top_k is an exact brute-force scan, so recall@k is 1.0 by construction and is kept as a correctness guard with a control (complete 1.000 / damaged 0.333 / empty undefined), not published as a headline. The number that bounds the design is cost, and it tracked op-log depth rather than live points.

✅ Dataset::supersede_key — a merge can now collapse superseded keys

Dataset::supersede_key(record) -> Option<&str> declares that a later record with the same key makes an earlier one semantically dead; merge_once then keeps, per key, only the maximum-sort-key record among its own sources. Every record it drops already lost a latest-wins fold to a record it keeps, so the answer cannot move.

row	sealed parts	stored records	superseded	query	µs/record
10x500ops (before)	9	4,608	0	65.02 ms	14.11
compacted, default policy	3	1,404	3,596	19.81 ms	14.11
compacted, tuned policy	1	500	4,500	7.13 ms	14.26

9.2x fewer stored records, 9.1x faster, converging to exactly the live set. Per-record cost is identical across every sealed row, so the cost model is linear in stored records and compaction removes precisely the right term.

⚠⚠ Which datasets may opt in — a claim about readers, not a performance switch

dataset	eligible	why
VectorDataset	✅ opted in, on point_id	live_points is its only read path
RuntimeDataset (D8)	❌ disqualified	RuntimeStore::watch_since returns the op log after a cursor — compacting it would delete the history a watcher resumes from
D1EventLog	❌ never	append-only and immutable by platform rule; replay is the source of truth
catalog attribute / relation folds	⬜ candidate	the catalog's own semantic call

One history reader disqualifies a dataset, even when its other read paths are latest-wins folds. Note index_key is not the compaction identity: for D6 that is the collection, and compacting by it would collapse a whole collection to one record.

⚠ Tombstones accumulate. A tombstone is the max-sort-key record for its key, so it is kept and a delete cannot be resurrected. They clear only when retention drops the part.

⚠ The default merge policy is wrong for a compacting dataset. Merge is one-level, and under the default compaction stalls at 1,404 records — not because small_part_max_records was exceeded but because only 3 small parts remained and trigger_run_len = 4 never fires again. Set the bound at or above the expected live-set size and a trigger that can be reached.

ehdb_l0_records_superseded is pinned at 0 and asserted to equal exactly the records dropped. A dataset declaring no supersede_key can never move it.

What is not measured yet

• p99, seal/merge/reclaim, memory, concurrent writers / multi-shard, ehdb-feed, queue depth, benches in CI — all done, above.
• The drain cost mechanism is measured and not attributed (ehdb#389).
• The networked transport is still hand-run. examples/dispatch_bench.rs attributes publish→claim across the real socket topology and nothing in CI executes it; the benches CI runs are the in-process drain primitives.
• Nothing measures a multi-node deployment. Every figure here is one process. Tail replication and writer election are specced, not built, so there is nothing to measure — and a single-process number must not be quoted as a distributed one.
• No ANN index exists, so recall@k is not yet a real measurement — it becomes one when an approximate index does.
• Fold determinism still has no engine-side enforcement point (A5): it is measured and proven order- and completeness-sensitive by test, but nothing in the engine refuses or counts a non-deterministic fold at runtime.

See also

• Measures: complete and measured — the acceptance criteria these results are held to.
• North star: distributed registry — the phase plan.
• noetl/ehdb#388 (lazy-ack O(n²)) and noetl/ehdb#389 (drain attribution + the #298 cap tradeoff) — the two open findings.