chore: add real session bundle measurement harness (#1336)

[Nyvo-io]

Refs #1336

Summary

This keeps the timeboxed work as a reproducible measurement harness and an evidence-status decision record. It does not implement the production Session Bundle activation or control plane.

• Accept sanitized real single-session exports instead of deriving release claims from synthetic state fixtures.
• Build and validate the target POSIX tar.zst layout with a versioned manifest, exact entry equality, byte counts, SHA-256 digests, and path-safety checks.
• Recursively exclude .git, node_modules, common credential files, and the source storage-root authority marker.
• Stream tar creation, compression, decompression, validation, and sequential manifest hashing to bound memory and file-descriptor use.
• Run fresh-process bootstrap through the real storage and Harbor-cell seams, resume the exported session, rebase restored paths to the materialized bundle, and record readiness before provider latency.
• Continue the child through clean turn completion so the early timing point cannot hide a failed restore.
• Keep provider TTFB optional and separate from local hydrate/bootstrap timing.
• Mark FakeBackend smoke runs as decisionReady=false; require a real sanitized corpus before reporting decision percentiles.

Current decision status

The document retains the storage/image/compression recommendations that are supported by implementation constraints, but it does not present the old synthetic 24/32 MiB limits or 250 ms budget as measured decisions. No real user session corpus is available in this checkout, so this PR intentionally leaves #1336 open.

Validation

• npm run format:check
• npm run lint
• npm run build:test
• npm run typecheck
• npm run test:scripts: 130/130 passed
• Session Bundle measurement tests: 14/14 passed
• Local bootstrap readiness timing test passed
• git diff --check

[Astro-Han]

This spike does not answer #1336 yet. The harness is reproducible, but the two measurements behind the decision record are not measuring the system the issue asks about.

The bundle distribution comes entirely from createSyntheticState(). The reported p99 is the one hard-coded 120-turn fixture, every artifact is generated compressible text, and the payload is a custom byte stream rather than the target tar.zst. --workspace only makes the workspace real; the session state remains synthetic. That means the 6.85 MiB p99, roughly 5 MiB of headroom, and the 24/32 MiB limits are fixture assumptions, not evidence from real coding sessions.

The cold-start number has the same problem. --child-boot starts the measurement script, reads and decompresses the state-only bundle, parses the first manifest line, then exits. It does not materialize the workspace or initialize the Maka runtime, stores, executor, or backend. The reported 71.11 ms is Node startup plus manifest parsing, not Maka activation, so it cannot support a 250 ms hydrate-plus-boot budget.

There are two smaller correctness issues. Directory filtering only classifies paths by their first segment, so nested node_modules and .git directories are included in the portable workspace. The OpenHands and SWE-agent source links in the decision record also return 404.

I would keep the harness as a synthetic smoke benchmark, but I would not close #1336 or turn these numbers into v1 limits yet. The spike needs sanitized real session exports, the target archive layout, and fresh-process activation through the closest real bootstrap with a fake backend. Until then, the document should call these candidate assumptions rather than measured decisions.

[Nyvo-io]

Implemented and reviewed in commit c7e91dc.\n\n- Replaced synthetic state samples with real sanitized session-export inputs and a real POSIX tar.zst archive (manifest, state, workspace).\n- Added fresh-process bootstrap through the real SessionStore, RuntimeEventStore, Harbor cell, and FakeBackend path.\n- Recursively excludes nested .git and node_modules entries, supports UTF-8 paths, validates manifest/path safety, archive-entry equality, byte counts, and SHA-256 digests.\n- Added defense-in-depth JSON/JSONL redaction without replacing schema fields such as tokenUsage. Smoke-only runs are explicitly marked decisionReady=false and cannot be used as percentile evidence.\n- Removed the invalid synthetic decision numbers and stale external links from the decision record.\n\nValidation: npm run build:test passed; measurement tests pass (2/2); Biome and syntax checks pass.

[Nyvo-io]

Rebased onto current main and updated in 01fdc9ca.

This revision addresses the measurement concerns without retaining the old synthetic conclusions:

• sanitized real single-session exports and the target POSIX tar.zst layout;
• recursive .git / node_modules / credential exclusion;
• manifest entry, byte-count, digest, and path-safety validation;
• fresh-process restore through real storage and Harbor-cell seams;
• resume of the exported session with materialized path rebasing;
• readiness measured at the first FakeBackend event while the child still completes and validates the turn;
• sequential manifest hashing and streamed archive processing;
• smoke evidence explicitly remains decisionReady=false.

The PR body and title now state the current evidence honestly: this supplies the real measurement harness, but does not claim release percentiles or close #1336 without a real sanitized session corpus.

Validation: format, lint, full workspace build/typecheck, script suite 130/130, measurement tests 14/14, and the first-event timing regression all pass.

[Astro-Han]

Approving because the remaining findings are non-blocking P2 follow-ups. One metadata issue cannot be attached inline: GitHub currently reports #1336 in this PR's closingIssuesReferences, despite the body saying that this harness does not close the issue. Please remove that closing linkage before merge so the issue remains open for the real-corpus measurement. I also left the timing-boundary issue inline.

[Astro-Han]

Five inline findings below. The P1 and timing-boundary findings affect whether a report can be treated as trustworthy decision evidence; the P3 covers archive write integrity.

[Astro-Han]

I did not find a P0 or P1 issue, so I am approving this PR. I found these follow-ups:

• [P2] A workspace may contain its paired session export. The state is then archived once as workspace data and again under state, while the report may still say decisionReady=true.
• [P2] The bootstrap sample does not prove a successful restore. It performs extra full reads before the production path and does not check whether the resumed invocation completed.
• [P2] decisionReady does not qualify the runtime environment. A local Darwin/Node 26 run can satisfy the flag even though the decision document requires a controlled Node 24 image.
• [P2] JSON and JSONL preparation uses several full-size in-memory copies. Concurrent workspace traversal also retains every entry list, so memory use is not bounded for the largest samples.
• [P2] The public resumeSessionId path can run with a stored header that disagrees with input.cwd or input.config.
• [P3] The script copies session-bundle policy constants and path rules from Storage, creating a second policy source.
• [P3] Tar creation does not detect a source file changing between stat and streaming.
• [P3] JSON sanitization pretty-prints compact production files, which biases the size measurement upward.

These do not need to block a measurement-only PR, but they should be resolved or documented before the harness is used to set production budgets.

Please push back if any input or environment above is intentionally unsupported. These comments should be checked against the intended measurement contract, not accepted by default.

[Nyvo-io]

Addressed the remaining review feedback in a53cf1d as one consolidated change.

• Rejects workspace/export root overlap and requires both smoke and restored invocations to complete; restored thinkingLevel is preserved through the probe.
• Splits bundle-size and bootstrap-latency readiness, requiring Node 24 on Linux x64/arm64 plus a non-empty runtime build identity for the combined gate; CLI help and the decision record now match.
• Streams JSONL sanitization, bounds plain JSON at 1 MiB, avoids retaining every paired workspace entry list, preserves unchanged JSON bytes, and safely reserializes duplicate-key JSON so discarded secret values cannot survive.
• Validates resume execution identity before backend registration or artifact writes, reuses Storage-owned bundle policy, and detects tar source-size changes.
• Existing coverage continues to prove per-sample workspace binding, the run-start timing boundary, the tar streaming path, and short-write retries.

Validation: CI typecheck, test, and e2e are green; local build, typecheck, lint, format, script tests (172/172), measurement tests (26/26), and serial runtime/CLI suites pass. Independent final review returned no findings. #1336 remains open (closingIssuesReferences is empty).

[Astro-Han]

Reviewed a53cf1d9; merged as 2dcfdee2b.

The earlier evidence gaps are closed. Overlapping roots are rejected, restore samples require a completed invocation, readiness checks the runtime environment, resume identity is validated before execution, and large inputs no longer require unbounded in-memory copies. CI was green. No P0–P3.