refactor(improvement): collapse optimization API onto agent-eval selfImprove

[drewstone]

What

One entry point for closed-loop optimization: agent-eval's selfImprove (@tangle-network/agent-eval/contract). It wraps runImprovementLoop + gepaDriver + the held-out gate + analyzeGeneration + production intake behind a single budget-shaped options object. agent-runtime keeps only the one genuinely runtime-specific piece — the CODE-surface ImprovementDriver (git-worktree mutation via CandidateGenerator), which you pass to selfImprove as driver.

Why

We had three overlapping optimization surfaces in this repo. optimizePrompt was a thin wrapper over runImprovementLoop; report-eval-runs re-implemented production-run intake. Both are now strictly subsumed by selfImprove + the /contract analysis helpers (analyzeRuns, partitionRunsByAuthoringModel). One function, not a wrapper zoo.

The unlock was a version skew: installed @tangle-network/agent-eval was 0.76, whose selfImprove lacked analyzeGeneration (the analyst→reflection wire we depend on). 0.83 adds it — so selfImprove is now strictly a superset of what optimizePrompt did, and the wrappers can go.

Changes

• delete src/improvement/optimize-prompt.ts (+ test) — subsumed by selfImprove.
• delete src/improvement/report-eval-runs.ts (+ test) — subsumed by selfImprove hostedTenant + /contract analyzeRuns / partitionRunsByAuthoringModel.
• migrate selfImproveLoopRunner (src/loop-runner.ts) and bench/src/improve-prompt.ts onto selfImprove. Field renames: baselineComposite→baseline.compositeMean, winnerComposite→winner.compositeMean, delta→lift, decision→gateDecision, prompt→winner.surface, rationale→winner.rationale; budget gains holdoutScenarios / reps / promoteTopK.
• trim src/improvement/index.ts to export only the CODE-surface driver pieces (improvementDriver, agenticGenerator, reflectiveGenerator).
• bump @tangle-network/agent-eval 0.76 → 0.83 (root + bench/); 0.83 is the first release whose selfImprove exposes analyzeGeneration.

No back-compat shim — this is greenfield optimization plumbing.

Verification

• −817 LOC net.
• root typecheck 0 / lint clean / build success / 658 tests pass.
• bench typecheck 0 (the improve-prompt migration compiles against 0.83).
• merges cleanly into main (rebased on top of docs(bench): unify rollout/shot terminology + honestly scope the HumanEval gate #170/refactor(bench): rename gepa-refine → improve-prompt (name by purpose, not method) #171; reconciled the gepa-refine → improve-prompt rename).

[tangletools]

✅ No Blockers — 7c0a1790

Readiness 72/100 · Confidence 95/100 · 7 findings (2 medium, 5 low)

	deepseek	glm	aggregate
Readiness	72	83	72
Confidence	95	95	95
Correctness	72	83	72
Security	72	83	72
Testing	72	83	72
Architecture	72	83	72

Full multi-shot audit completed 7/7 planned shots over 8 changed files. Global verifier still owns final merge decision. | Full multi-shot audit completed 7/7 planned shots over 8 changed files. Global verifier still owns final merge decision.

🟠 MEDIUM Stale +20pp win claim contradicts repo's own evidence ledger — bench/src/improve-prompt.ts

Line 13: 'We proved evidence-gated refinement beats blind (FinSearchComp +20pp) with a HAND-WRITTEN refine directive.' Per CLAUDE.md (repo root): 'The earlier +20pp steering proven was confounded compute — a cautionary precedent.' The claim is demonstrably stale and misleads anyone reading this file as user-facing documentation of what the bench proved. The PR touched the adjacent comment block (lines 1-4) so this file is in scope. Fix: update the comment to reflect the actual evidence state (the +20pp was confounded; subsequent contr

🟠 MEDIUM peerDependencies range too wide — code requires >=0.83.0 — package.json

DevDependency @tangle-network/agent-eval was correctly bumped from ^0.76.0 to ^0.83.0 (line 104), but peerDependencies (line 127) still declares >=0.76.0 <1.0.0. The new code in src/loop-runner.ts:29 imports selfImprove, SelfImproveOptions, SelfImproveResult from @tangle-network/agent-eval/contract — a subpath export added between 0.76.0 and 0.83.0. A consumer with agent-eval 0.76.0–0.82.0 would get a module-resolution or import error at runtime/typecheck. Fix: tighten peerDependencies to >=0.83.0 <1.0.0.

🟡 LOW backstop timer not unref'd — keeps event loop alive — bench/src/humaneval-gate.mts

Line 166: setTimeout(() => finish({ pass: 0 }), dockerTimeoutMs + 3000) — the backstop timer is cleared on normal/error paths via clearTimeout(backstop) inside finish/fail, which is correct. However, the timer is not .unref()'d, so while the pool workers are running, an idle backstop timer will prevent Node from exiting early. In practice this is harmless (the pool awaits all workers), but .unref() would be marginally cleaner for a bench script that might add a top-level timeout later.

🟡 LOW docker rm -f cleanup is fire-and-forget with no error logging — bench/src/humaneval-gate.mts

Line 147: execFile('docker', ['rm', '-f', name], () => {}) — the empty callback silently swallows any error from docker rm -f. If docker itself is down (the case where the daemon is unreachable), this will fail silently on every cleanup. Not a bug (the container won't exist if docker run failed), but adding if (err) console.warn(...) would aid debugging stuck-container issues in CI.

🟡 LOW Dropped seed: 42 from optimizePrompt→selfImprove migration — bench/src/improve-prompt.ts

The old optimizePrompt call passed seed: 42 (line 576 in the old file). The new selfImprove call has no seed field. If selfImprove uses a nondeterministic seed by default, this changes GEPA's generation-to-generation reproducibility. Verify that selfImprove's default seed behavior matches, or add a seed field if the new API supports it. Impact: bench reproducibility only, not production.

🟡 LOW Breaking export removal: optimizePrompt and reportOptimizationRun — src/improvement/index.ts

Removes re-exports for optimizePrompt, reportOptimizationRun, OptimizePromptOptions, OptimizePromptResult, OptimizationRunMeta, optimizePromptResultToEvalRunEvents, and OptimizePromptReflection from the public barrel. All internal consumers have been migrated to @tangle-network/agent-eval/contract (loop-runner.ts, bench/src/improve-prompt.ts). The package.json version (0.44.0) should be bumped to reflect this breaking change for any external consumer importing from @tangle-network/agent-runtime/improvement.

🟡 LOW selfImproveLoopRunner ignores AbortSignal — src/loop-runner.ts

Line 285: return async () => selfImprove(...) discards the signal: AbortSignal parameter from the DelegatedLoopRunner type. This is pre-existing (the old optimizePrompt wrapper had the same shape) so not a regression, but callers passing an AbortSignal get no cancellation semantics. Fix: thread signal into selfImprove options if the substrate supports it.

---

_{tangletools · 2026-06-06T00:57:00Z · trace}

[tangletools]

✅ Approved — 7 non-blocking findings — 7c0a1790

Full multi-shot audit completed 7/7 planned shots over 8 changed files. Global verifier still owns final merge decision. | Full multi-shot audit completed 7/7 planned shots over 8 changed files. Global verifier still owns final merge decision.

Full immutable report for this review: trace

Summary comment for this run: full summary

---

_{tangletools · 2026-06-06T00:57:00Z · immutable trace}