Experiment benchmarks performance frontier lab plan

Historical repository record migrated on 2026-10-10. Original repository path: docs/experiments/benchmarks/performance-frontier-lab-plan.md Original SHA-256: e4bac0870a3992df096779fe9f0d0f23f94dcdebd1bd21c453fe4c0d99133d57 Stated outcomes, measured revisions, failures and limits remain historical. Local raw evidence is private and is not uploaded. Relative links and the data-drive prefix were normalized.

Performance frontier: next lab queue

Status: setup; no new performance results. Prepared 2026-10-05 after source inspection and the completion campaign. This is a prospective planning record, not a registered measurement protocol. Existing protocols and their failures remain unchanged. Follow the standing orchestration policy (local checkout reference docs/experiments/lab-orchestration.md; not present at origin/main).

The owner's subsequent map/ownership/Rust direction is developed in the storage catalog lab extension (local checkout reference docs/experiments/benchmarks/storage-catalog-rust-lab-plan.md; not present at origin/main), within this campaign's remaining budget. Existing outcomes and registered gates stay in their original records.

Admission attempt 01 subsequently failed closed with exit 2 because the required data-drive scratch is read-only. It also records binary-provenance and campaign-accounting prerequisites found in review.

Landscape and objective

Preserve exact telemetry and bounded memory while reducing total work per accepted and queried byte in development and small deployments. Optimize the whole service, including drain and recovery, rather than builder time alone.

Evidence	Interpretation	Remaining question
256 MiB builder: reference 1250.45 MiB versus bounded 39.74 MiB incremental heap; median 5.563 versus 5.335 seconds	The measured builder has bounded heap at roughly reference speed	Spill traffic, concurrent ingestion, mixed row shapes and process-wide limits
Warm broad-tail query: Scan about 171 MiB versus Walk 34 MiB peak heap; both about 112 ms under instrumentation	Retaining fewer allocations alone does not explain execution cost	Decode, selection, result construction, serialization and page size
Twenty-node mixed restart cell recovered 300000 logs, 18000 spans and 3903 ACKed Batches	Finite mixed integration works on that snapshot	Sustained behavior and actual service caps, separate from the grader's cgroup
Large rows read 9156 versus 20350 rows in narrow windows, but failed strict pruning equivalence	Physical grouping changes the cost surface	Does the gain survive other windows, compression costs and broad queries?

These are historical, fixture-specific measurements. Heap, RSS, cgroup memory and cumulative allocated bytes are distinct. The existing Scan default remains.

The implemented path is Spindle collection/local Spool → TLS delivery → bounded server submission queue → grouped two-sync journal commit/ACK → asynchronous bounded sort/Parquet publication → checkpoint/reclaim. Queries already read pending journals; publication is not a prerequisite for query availability.

Source anchors:

• Spindle: collection, trace commits and delivery share runtime scheduling.
• Store: 64 MiB queued-byte bound, 1 MiB group target and custody ordering.
• Sealer: background workers share CPU/disk with intake and queries.
• Bounded builder (local checkout reference crates/fabric-server/src/segment/bounded.rs; not present at origin/main): 16 MiB/32768-row runs, sixteen-way merge, byte-capped output groups.
• Query: Scan decodes journal Groups before filtering; Walk selects indexed sources, with shared state behind a mutex.
• Verification matrix: HIST-7 concurrency, HIST-9 restart/block growth, SPOOL-1 disk pressure remain consequential gaps.

Three competing explanations guide the experiments: CPU spent reconstructing answers, IO spent converting/re-reading data, and scheduling/queue contention. Smaller journals might reduce pending query work but increase publication and metadata overhead. Larger sort buffers might remove a merge pass but compete with queries for RAM. Faster individual queries may still serialize at source discovery. These interactions must be measured, not assumed.

Labs and accountability

Lab	Accountable PI for this setup	Owned packet	First decision
Capacity and lifecycle	capacity_landscape, GPT-6.1-SOL medium	capacity packet	Is spill/merge IO worth optimizing before more scheduling work?
Query and availability	query_landscape, GPT-6.1-SOL medium	query packet	Which query phase still dominates after heap reduction?
Recovery and operations	operations_landscape, GPT-6.1-SOL medium	operations packet	Does the candidate remain useful inside real service budgets?

Root owns admission, source snapshots and synthesis. PIs review evidence; their agreement is not a correctness gate. Luna assistants are dispatched only for bounded fixture inventories, accounting or receipt audits once a slot is free. They do not duplicate PI analysis or independently launch workloads. Labs are persistent responsibilities, not three continuously running agent processes.

Queue and admission

Order / ID	Owner	State	Dependency / outcome required
0 / F0	Root + operations	preparing	Inventory dirty source and old receipts; retain failed runs; verify prepared harnesses and archive ownership
1 / C5	Capacity	proposed	Existing spill observer validation and exact output checks; then register a merge-boundary investigation
1 / Q3	Query	proposed	Reuse corrected complete-chain profiling; register phase/page-size experiment
1 / R3	Operations	proposed	Compile and execute queued storage/publication cases under their existing protocols before relying on them
2 / Q4	Query	proposed	Q3 attribution; publication checks; register concurrency and restart-cost experiment
2 / C6	Capacity	proposed	C5/Q3; register occupancy × query-demand screen with one sealer worker
2 / R4	Operations	proposed	R3 and nested-cgroup verification; existing capped ordinary/pressure protocol, then matched-demand attribution
3 / F1	Root	deferred	Independent confirmation of at most one candidate per bottleneck after screens

Independent diagnostic work need not wait for full builder acceptance. Existing dependent soak/outage/lifecycle acceptance remains blocked by the recorded C2 failure according to its own protocol. A prospective physical-layout decision needs a separate policy/protocol commit; it cannot turn the old failure into a pass. Installation and release qualification remain separate work.

F0 must distinguish executed storage tests from subsequently added uncompiled cases, and completed mixed integration from unrun pressure tests. Reuse completion tooling (local checkout reference tools/bench/labs/completion; not present at origin/main) rather than create another runner. Preserve each command, exit, working-tree snapshot and protocol copy. Existing run records are not edited to imply new execution.

Measurements and experimental discipline

Every executable protocol must freeze the fixture/seed, binary hashes, exact commands, varied parameter, offered query schedule, baseline, output limits and measurement boundaries before launch. Start from existing development/small fixtures; do not infer an unspecified customer workload. Use three fresh alternating baseline/candidate pairs for confirmation. Single screening cells nominate hypotheses only. Cache state and process restart are separate factors; restarting a process does not produce a cold OS cache.

Record CPU seconds and CPU per accepted MiB; server/node RSS/HWM; incremental heap and cumulative allocation in diagnostic variants; cgroup anon/file/kernel, pressure/throttling/OOM; disk read/write and logical spill bytes; scratch peak; build/publication duration; queue bytes/oldest age and drain time. Report local Spool accepted Batches/s AND bytes/s, rejected attempts, SDK accepted exports, server durable ACKs and retransmits separately. Batch size can change without a throughput change. Keep offered, accepted and recovered counts reconciled.

Report query p50/p95/p99, samples, errors and throughput by kind, first page and complete chain separately. Keep fixed offered demand to expose queue growth. Measure Spool-commit → server ACK → first observed query visibility with stable clock evidence and polling bounds; do not call a local HTTP200 a server ACK. The suggested 30-second headroom does not weaken existing freshness contracts.

Use plain binaries for primary performance timing and counted/probed variants for attribution. Do not sum inclusive nested phases. Grade complete answers with the unchanged independent Python oracle; retain missing/corrupt optional-index fallback, tie ordering, snapshot pagination and expected-gap controls. New checkers need a representative defect they demonstrably reject.

Draft confirmation decision: at least 10% lower primary CPU or wall cost in each of three matched pairs, no exactness failure, no more than 5% regression in the other primary cost, p99 query/ACK latency or peak memory, and no growing backlog. Register the precise estimands and minimum sample counts for each packet before measurement. Missing samples, clock discontinuity or uncompleted drain make a cell inconclusive. A trade-off that fails this rule can remain documented as an operating-region option; it is not a default-change recommendation.

Resource and token admission

Preparation allowance: estimated 18000 agent tokens (three targeted PI packets plus coordinator synthesis), with a separate 4500-token contingency. These are planning estimates, not enforced or observed billing. At each handoff reconcile available telemetry; do not launch an assistant merely to fill a slot.

Initial execution proposal: four aggregate workload-hours, at most 90 minutes per lab, within that total, including builds/checks. Stop at budget exhaustion; reassess before allocating confirmation. No individual job exceeds the default 30-minute deadline. No long soak is hidden in this allowance.

Serialize all builds, tests, validators and experiments through python3 tools/resource_group.py -- COMMAND ...: 16 GiB high, 20 GiB max, zero swap. Keep scratch/build caches under the mounted $FABRIC_DATA; require 16 GiB free, cap owned scratch at 8 GiB/job and retained evidence at 256 MiB/lab unless separately reviewed. Application child limits are additional, never substitutes for outer containment. Measure actual service limits; distinguish experimental development caps from installed small-profile limits and CPU weights from hard quotas.

Stop on custody/query mismatch, unexpected OOM, unenforced limits, disk floor, clock failure or uncontrolled queue growth. Preserve failure evidence before byte-verified cleanup; clean successful owned fixtures and transient processes. Do not remove shared caches or another run's scratch. No remote workload is queued; the small remote is not a substitute for a supported deployment host.

Current execution constraint: this session's filesystem policy permits workspace edits but not writes to the required data drive, and Git metadata is read-only. Consequently no workload, validator, registration commit or new performance measurement was attempted during setup. Do not bypass this via a service manager or substitute /tmp. Resume admission in an environment that allows the prescribed data-drive/cgroup workflow.

Synthesis checkpoint

For each packet publish observations, rejected/null hypotheses, source-backed mechanism, competing explanations, resource cost, cleanup and uncertainty. Root compares total ingestion + query + sealing CPU, not isolated winners. Escalate only the strongest unexplained interaction into the exploration reserve. Candidate implementation follows evidence and keeps contracts/defaults intact; wire-format, durability, oracle and qualification changes require their own scope.