feat(experiment): sealed experiments — the registered rule is the executed rule

[drewstone]

What

Adds the @tangle-network/agent-eval/experiment subpath: experiments as sealed objects (charter Wave 2).

Rules are data. A 14-family typed AST carries every registered rule — predicates, admission funnels with registered waivers, deterministic selections with a closed read set, estimands, interval specs with seeds, ordered decision tables (and report-only for registrations that decline a threshold), obligations, validity gates, halt rules, budget schedules with named ledgers, matched-budget rules, and reissue policies. sealExperiment canonicalizes and hashes the whole tree; openSealedExperiment verifies the digest and returns the only execution surface. No executor takes a parameter for alpha, threshold, metric, or stopping rule, so registered-vs-ran drift is unrepresentable rather than checked. An amendment is a re-seal with a {at, reason, blind[], digest} chain.

New objects close the four measured gaps; everything statistical underneath is re-exported from its existing home (statistics.ts, clustered-paired-binary.ts, sequential.ts, the campaign gates, paired-arms.ts, pre-registration.ts, experiment-tracker.ts):

• clusteredPower — whole-cluster percentile-bootstrap power simulation with a refusal verdict inside the artifact. The closed-form floor refuses before simulation: with C clusters the exact sign-flip test can never produce a two-sided p below 2^(1-C), so the {6,3,3,2} structure (0.125) and any 3-cluster design (0.25) are refused at alpha 0.05 at ANY effect size — the burn that was previously learned by running the experiment.
• buildFunnel / executeAdmissionRule — the denominator chain as a first-class object: refuses a stage that gains rows, named exclusions that do not sum, and partition overdraw; composes across boundaries; renders itself (text table + JSON); partitions are pooling: 'never'.
• verifyMatchedBudgets — realized-token arm matching under the registered tolerance, returning a verdict whose refusal carries onFail: 'refuse-contrast'.
• defineExperiment — cross-reference validation: a decision condition can only read a registered interval, estimand, or obligation; a halt rule only registered gates; a filter-of base only a registered selection or sealed subset.

Acceptance — the three preregistrations

tests/experiment/preregistration-acceptance.test.ts expresses the week's three hand-written preregistrations with the module, seals them, and reproduces each recorded decision by executing the sealed rules against recorded evidence (vendored with provenance in recorded-fixtures.ts):

fixture	recorded decision reproduced
killtest-20260810	all four gates FAIL on recorded evidence (rep-4 oracle flip 1/15; 2 changed rows, both largest-eigenval; zero-call control; power ceiling 0.692 < 0.80) → halt fires refuse-spend ($0.00, contrast never run); positive interval without the registered control lands in blocked-pending-registered-control
freelunch-20260810	admission funnel reproduces 48>43>35>35>32 + secondary 3; uniform-pass budget reproduces uniform n=2; the amendment-6 ledger under the SAME sealed rule refuses pass 2 (n=1) — the registered-vs-ran drift the seal makes unrepresentable; report-only estimands reproduce 3/64 and 2/32; identity gate aborts deepseek-v3.2→v4-flash and passes glm-5.2
tbench-20260808 m2	round-robin selection reproduces the recorded 20 rows in pick order; m3 = certified-task filter of the SEALED m2 draw (16/16 in order); decision table on the recorded interval (−0.100, +0.567) → not-certified-at-this-n

Checks

• pnpm typecheck — clean
• pnpm test — 386 files passed, 5341 tests passed, 0 failed (3 pre-existing skips)
• pnpm build — clean; dist/experiment/index.js smoke-imported, refusal fires from the built artifact
• pnpm verify:package — exit 0 (publint + attw + packed-exports resolution incl. ./experiment)
• npx biome check src/experiment tests/experiment — clean

The analyst-benchmark dependency-lock digest is bumped because package.json gains the ./experiment export (same flow as #553).

Docs: docs/experiment.md (the covenant + fold-in map), cross-linked from the charter's Wave 2 line and CLAUDE.md's doc map.

[tangletools]

✅ Auto-approved drewstone PR — f438a176

This PR was opened by the trusted drewstone account.
The full PR reviewer audit still runs separately and will publish findings if it detects issues.

_{tangletools · auto-approval · reason: drewstone_author · 2026-08-10T20:12:50Z}

[tangletools]

🟡 Value Audit — sound-with-nits

Verdict	sound-with-nits
Concerns	2 (1 low, 1 weak-concern)
Heuristic	0.0s
Duplication	0.0s
Interrogation	135.2s (2 bridge agents)
Total	135.2s

💰 Value — sound-with-nits

Adds a sealed-experiment subpath where every registered rule is a hashed, typed AST node and the only execution surface takes evidence alone — coherent, deeply evidence-backed, and composes existing stats rather than forking them; one genuine overlap with trace-repair's denominator chain is the only

• What it does: Adds the @tangle-network/agent-eval/experiment subpath (src/experiment/{ast,define,funnel,budget,power,index}.ts). Experiment rules become a 14-family typed data AST (predicates, admission funnel stages+waivers, deterministic selections with a closed read-set, estimands, interval specs with embedded seeds, ordered decision tables plus a 'report-only' variant, obligations, validity gates, halt rule
• Goals it achieves: Make the registered rule and the executed rule the same object, so design drift (changing alpha, swapping the metric, dropping a stopping rule after peeking) is impossible rather than audited-after-the-fact. Move every refusal — an underpowered cluster count, a non-monotone funnel, a mismatched arm budget, a non-total decision table — inside a typed verdict artifact instead of prose beside a numbe
• Assessment: Sound and unusually well-evidenced. The grain matches the codebase exactly: typed data nodes over closures, typed outcome objects with refusals inside them, content-hash sealing, and fail-loud ValidationError/CaptureIntegrityError throws — all established patterns (pre-registration.ts, clustered-paired-binary.ts, campaign/gates/*). The acceptance suite (tests/experiment/preregistration-acceptance.
• Better / existing approach: One genuine overlap exists. The new buildFunnel (src/experiment/funnel.ts:67, assertFunnelReconciles at :142) and the pre-existing buildDenominatorChain / assertChainReconciles (src/trace-repair/admission-records.ts:283 / :340) implement the identical monotone-reconciliation invariant (input = surviving + excluded; throw on a stage that gains rows or a chain that doesn't add up). docs/experiment.m
• Model: opencode/zai-coding-plan/glm-5.2
• Bridge attempts: 2
• Bridge warning: opencode/kimi-for-coding/k2p7: opencode: opencode error

🎯 Usefulness — sound

A coherent sealed-experiment substrate that composes the existing statistics and pre-registration primitives, closes four named charter gaps, and is validated by a real three-preregistration acceptance suite — landing ahead of its first production caller (Wave 3 runtime wiring), which is the expecte

• Integration: Fully wired and reachable. The ./experiment subpath is registered in package.json:142-146 and scripts/build-entries.mjs:27. All 14 AST node families have interpreters routed through the single execution surface RegisteredExperiment (define.ts:379-502): decide, admit, select, gate, halt, runUniformPassBudget, projectNLadderBudget, matchedBudgets, estimate, interval. No produ
• Fit with existing patterns: Composes, does not compete. (1) Sealing reuses the existing hashJson/canonicalize from pre-registration.ts:19,113 — the same primitive signManifest is built on — extending the established single-metric pattern to a full-design registration. docs/experiment.md:98 explicitly positions HypothesisManifest as the lightweight form and sealExperiment as the full form; clear layering, no duplica
• Real-world viability: Holds up past the happy path. The acceptance suite is the charter's demanded kill test: three real preregistrations reproduced, including the registered-vs-ran divergence the seal makes unrepresentable (freelunch amendment-6 ledger) and the $0.00 halt (killtest four-gate failure). Edge inputs are fail-loud everywhere: zero rows throw (ast.ts:297,322), empty intersect throws (ast.ts:240), non-total
• Model: opencode/zai-coding-plan/glm-5.2
• Bridge attempts: 1

🔎 Heuristic Signals

🟡 Cruft: magic number added tests/experiment/budget-and-seal.test.ts

•  resamples: 1000,
  

💰 Value Audit

🟡 Two monotone denominator-chain implementations now coexist [maintenance] ``

src/experiment/funnel.ts:67 (buildFunnel) and src/trace-repair/admission-records.ts:283 (buildDenominatorChain) both enforce 'input = surviving + excluded, throw on non-monotone stage, throw on non-reconciling chain' — the core invariant is duplicated. The new one is generic (arbitrary stage ids, partitions, composition, text render); the trace-repair one is domain-bound (closed AdmissionExclusionReason enum, strata, versioned artifact). docs/experiment.md:99 declares the split intentional. No a

---

What this audit checks

It judges the change on its merits — not whether it was tasked out in an issue. Unticketed, fast-moving work is fine; the question is whether the change is good and whether a better or existing approach should be used instead.

Pass	What it asks
Heuristic	Vague title? Whitespace-only or cruft-bearing diff? (content signals only)
Duplication	Do added function/class names already exist elsewhere in the repo?
Value Audit	What does it do? What goal does it achieve? Is it good? Better architecture or already-exists?
Usefulness Audit	Does it integrate and fit? Will it hold up in real use and actually get used?

Findings are concerns, not blocks — the human reviewer decides what to do with them.

_{value-audit · 20260810T201711Z}