benchmarks: add indexed-write (ST-2), ttl-churn (ST-1), concurrent-rw (ST-5)

[kriszyp]

Summary

Three single-node storage benchmarks addressing the performance/benchmark gaps from §6.3 of the Harper Release Testing Strategy and §5 of the v5 Integration Test Plan (ST-1, ST-2, ST-5).

All three use the same @harperfast/integration-testing boot pattern as the existing YCSB harness, emit a single machine-parseable RESULT line for future regression gating, and run in ~30-60 s at their default --scale=quick.

---

ST-2 — Indexed-write throughput (benchmarks/indexed-write/)

Measures write ops/sec on three table variants that differ only in how many @indexed secondary fields they carry (0 / 3 / 5). Reports absolute ops/sec per variant and a ratio vs. the unindexed baseline so a regression gate can track relative index-maintenance cost across releases independent of hardware.

Scale	Records	Concurrency	Approx. runtime
quick (default)	5 000	16	~30 s (3 variants)
nightly	1 000 000	64	~15 min

node benchmarks/indexed-write/run.mts
node benchmarks/indexed-write/run.mts --scale=nightly

Parseable output:

INDEXED_WRITE_RESULT variant=baseline ops_per_sec=NNN
INDEXED_WRITE_RESULT variant=indexed3  ops_per_sec=NNN ratio_vs_baseline=N.NNN
INDEXED_WRITE_RESULT variant=indexed5  ops_per_sec=NNN ratio_vs_baseline=N.NNN

---

ST-1 — TTL-churn / map-size growth (benchmarks/ttl-churn/)

Sustained insert-with-TTL workload; samples the on-disk data directory size every --sample-every seconds. Asserts (and reports) that final storage size is ≤ 150% of the halfway-point size — i.e. that TTL eviction + compaction keeps the store bounded. The quick scale runs for 30 s (TTL is 60 s, so records are still live — this proves the insert path works). The nightly 30-min scale lets records expire and verifies compaction reclaims the space.

Do not run --scale=nightly locally (30+ min).

Scale	Records/wave	TTL	Duration	Sample every
quick (default)	10 000	60 s	30 s	5 s
nightly	1 000 000	60 s	1 800 s (30 min)	60 s

node benchmarks/ttl-churn/run.mts
node benchmarks/ttl-churn/run.mts --scale=nightly   # CI only

Parseable output:

TTL_CHURN_SAMPLE elapsed_s=NNN dir_bytes=NNN records_inserted=NNN
TTL_CHURN_RESULT duration_s=NNN peak_bytes=NNN final_bytes=NNN total_inserts=NNN bounded=true|false

---

ST-5 — Concurrent read+write (benchmarks/concurrent-rw/)

Seeds a highly-indexed table (5 @indexed fields: category, region, status, priority, tag), then runs N concurrent readers (multi-condition REST queries across those fields) and M concurrent writers (inserts) simultaneously for --duration seconds. Reports read p50 / p95 / p99 / max and checks p99 against a configurable ceiling (--p99-ceiling-ms, default 200 ms).

Scale	Seed records	Duration	Readers	Writers
quick (default)	2 000	15 s	4	2
nightly	200 000	120 s	16	8

node benchmarks/concurrent-rw/run.mts
node benchmarks/concurrent-rw/run.mts --scale=nightly

Parseable output:

CONCURRENT_RW_RESULT read_ops=NNN write_ops=NNN read_p50_ms=N.N read_p95_ms=N.N read_p99_ms=N.N p99_ceiling_ms=NNN ceiling_ok=true|false

---

Sample small-run output (on this dev machine, --scale=quick)

Indexed-write:

ST-2 Indexed-write throughput — RESULTS
variant          ops/sec     ratio    errors
baseline            6104         —         0
indexed3            6672     1.093         0
indexed5            8348     1.368         0

INDEXED_WRITE_RESULT variant=baseline ops_per_sec=6104
INDEXED_WRITE_RESULT variant=indexed3 ops_per_sec=6672 ratio_vs_baseline=1.093
INDEXED_WRITE_RESULT variant=indexed5 ops_per_sec=8348 ratio_vs_baseline=1.368

(Ratios > 1 at 5 k records are within variance — at 1 M records the index overhead becomes the signal.)

TTL-churn:

ST-1 TTL-churn / map-size growth — RESULTS
Total inserts : 268,064 (0 errors)
Peak dir size : 0.81 MiB
Final dir size: 0.77 MiB
Storage bounded: YES

TTL_CHURN_RESULT duration_s=30 peak_bytes=847882 final_bytes=811392 total_inserts=268064 bounded=true

Concurrent R+W:

ST-5 Concurrent read+write — RESULTS
Read ops     : 4,131 (0 errors)
Write ops    : 1,243 (0 errors)
Read latency : p50=11.5ms  p95=34.1ms  p99=49.9ms  max=120.1ms
p99 ceiling  : 200ms — OK

CONCURRENT_RW_RESULT read_ops=4131 write_ops=1243 read_p50_ms=11.5 read_p95_ms=34.1 read_p99_ms=49.9 p99_ceiling_ms=200 ceiling_ok=true

---

Test plan

• npm run build succeeds in worktree
• node benchmarks/indexed-write/run.mts runs end-to-end, 0 errors, prints INDEXED_WRITE_RESULT lines
• node benchmarks/ttl-churn/run.mts runs end-to-end, 0 errors, prints TTL_CHURN_RESULT bounded=true
• node benchmarks/concurrent-rw/run.mts runs end-to-end, 0 errors, prints CONCURRENT_RW_RESULT ceiling_ok=true
• CI: wire --scale=nightly variants into a nightly workflow (follow-up)
• CI: add regression-gate script that greps *_RESULT lines and diffs vs. stored baseline (follow-up)

---

Generated by a Claude Sonnet subagent (claude-sonnet-4-6) via Claude Code.

🤖 Generated with Claude Code

[gemini-code-assist]

Code Review

This pull request introduces three new single-node storage and throughput benchmarks for Harper: ST-1 (TTL-churn), ST-2 (Indexed-write), and ST-5 (Concurrent R+W), complete with schemas, configurations, and TypeScript runners. Feedback on the implementation highlights two critical issues: first, sorting the latency array inside the progress reporter in the concurrent read/write benchmark blocks the event loop and artificially inflates latency measurements; second, the recursive dirBytes function in the TTL-churn benchmark contains a race condition on the total variable that can lead to under-reported directory sizes.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[claude]

Review — PR #1379 (ST-1/ST-2/ST-5 benchmarks)

One non-blocking accuracy issue; no blockers.

Finding: totalInserts overcounts errors in ttl-churn (non-blocking)

In benchmarks/ttl-churn/run.mts line 331, totalInserts += batchSize runs unconditionally after each batch regardless of how many in that batch failed. Failed requests increment insertErrors, but totalInserts absorbs them too. The machine-parseable RESULT line therefore emits total_inserts that may exceed actual successful inserts.

The concurrent-rw benchmark does this correctly — writeOps++ is only inside the try block. The ttl-churn benchmark should align with that pattern.

Inline comment posted at the relevant line.

No issues with:

• dirBytes race: fixed in a976974 (sequential accumulation via reduce over Promise.all)
• Live p99 sort on the event loop: correctly removed from concurrent-rw; final sort happens once after all workers settle
• npm run build || true in the new workflow: follows the established pattern in integration-tests.yml for this repo's current type-error state; the "no tsc||true" rule applies to core library CI, not benchmark runners
• Workflow trigger safety: schedule + workflow_dispatch only (no pull_request), protecting the self-hosted runner from untrusted PR code
• Input handling: SCALE passed via env: not ${{ }} shell interpolation — avoids script injection
• bounded heuristic: logic and the false-positive note in the README are both correct
• TypeStrip compat: .mts files use only erasable syntax; no enums or namespaces

[kriszyp]

Addressed both findings from the Gemini/Claude review — thanks for the sharp catches!

benchmarks/ttl-churn/run.mts — lost-update race in dirBytes (commit a976974)
The shared total += accumulator inside concurrent Promise.all callbacks was a classic lost-update: each async callback could read and overwrite total before another finished, silently dropping sizes. Fixed by having each entry return its own size and summing with reduce after Promise.all settles — no shared mutable state across awaits.

benchmarks/concurrent-rw/run.mts — O(n log n) sort in the hot progress interval (same commit)
The 5 s ticker was calling readLatenciesMs.slice().sort(...) on the full growing array, blocking the event loop that drives the reader/writer workers — coordinated omission that would inflate the very p99 it was displaying. Dropped the live p99 from the interval (kept elapsed/remaining/counts); the final Report section still computes all percentiles correctly on the settled array, outside the hot path. Added a comment explaining the omission.

— Claude (Sonnet 4.6)

[claude]

totalInserts is incremented by the full batchSize even when some requests in the batch failed (failures increment insertErrors but totalInserts still counts them). This means the machine-parseable RESULT line emits a total_inserts that can be higher than actual successful inserts. In concurrent-rw, writeOps is incremented only inside the try block, which is the correct pattern.

Suggestion: deduct errors from the batch count, or count successes instead:

Suggested change

	totalInserts += batchSize;
	totalInserts += batchSize - (/* track per-batch errors */ 0);

A minimal fix: change to count only the non-error inserts. For example, accumulate a per-batch error counter and do totalInserts += batchSize - perBatchErrors. Alternatively, align with concurrent-rw's pattern by counting successes individually inside each promise.