feat(improvement): expose verified candidate populations

[drewstone]

Outcome

improve() now returns every verified optimizer candidate, not only the winner. Runtime joins Eval callback observations with the official GEPA graph by candidate digest, then returns each candidate as an exact AgentProfile plus Interface diffs or an explicit refusal.

Ownership

• Eval owns verified observation and GEPA graph artifacts.
• Runtime uses the existing profile materializer and Interface identity functions.
• Callback-only candidates retain explicit unavailable lineage.
• Duplicate GEPA graph nodes remain attached to one unique candidate.

Proof

• pnpm test: 2,451 passed, 6 skipped
• focused improvement tests: 51 passed, 2 skipped
• pnpm run typecheck
• pnpm run lint: 562 files
• pnpm run verify:package
• pnpm run check:version-bump: Runtime 0.131.8 and Bench 0.8.3 accepted
• independent read-only review requested

Release

This PR prepares Runtime 0.131.8 and Bench 0.8.3. It requires Eval 0.144.11.

[tangletools]

✅ Auto-approved drewstone PR — bd58730e

This PR was opened by the trusted drewstone account.
The full PR reviewer audit still runs separately and will publish findings if it detects issues.

_{tangletools · auto-approval · reason: drewstone_author · 2026-08-11T04:43:19Z}

[tangletools]

🟢 Value Audit — sound

Verdict	sound
Concerns	0 (none)
Heuristic	0.0s
Duplication	0.0s
Interrogation	129.2s (2 bridge agents)
Total	129.2s

💰 Value — sound

improve() now returns every verified optimizer candidate (joined across Eval's callback-observation and GEPA-graph artifacts by digest, each materialized as an exact profile + Interface diffs or an explicit refusal) and binds the winner to the GEPA bestIndex — a clean, in-grain composition with no e

• What it does: Adds result.candidatePopulation to ImproveMethodResult. After a method run, Runtime reads Eval's two provenance artifacts (readExternalOptimizerObservationArtifact, readGepaCandidatePopulationArtifact), dedupes candidates by candidateDigest, attaches callback-observation provenance and GEPA lineage nodes (parent indices, aggregate + selection scores) to each, then materializes each uniqu
• Goals it achieves: (1) Expose the full verified candidate population, not just the winner, so a downstream consumer can inspect, rank, or activate any candidate the optimizer actually evaluated — previously only result.candidate (the single winner) was available. (2) Make the method's winner accountable to its own verified evidence: the fix-commit binding prevents a method from declaring a winner that its GEPA gra
• Assessment: Sound and in the grain. The change composes three layers without duplicating any of them: Eval owns and reads the artifacts (via its own exported readers); Runtime owns turning a candidate surface into a complete profile by reusing the existing createProfileCandidateMaterializer (profile-surface.ts:285) — passed in as materializeProfile, not re-implemented; Interface owns identity via the exis
• Better / existing approach: Searched src/ for any existing candidate-population or optimizer-enumeration concept: grep'd candidatePopulation|gepaCandidatePopulation|readGepaCandidatePopulationArtifact (all matches are new in this PR), and reviewed src/intelligence/improvement-surfaces.ts which derives AgentProfileDiffs from candidate bundles for the activation layer — a different concern (sealing an already-chosen ca
• Model: opencode/zai-coding-plan/glm-5.2
• Bridge attempts: 2
• Bridge warning: opencode/kimi-for-coding/k2p7: opencode: opencode error

🎯 Usefulness — sound

Exposes every verified optimizer candidate (not just the winner) by joining Eval's callback observations with the GEPA candidate graph by digest, then materializing each into an exact profile+diffs or an explicit refusal — built entirely from existing Runtime/Interface primitives, with no competing

• Integration: The new candidatePopulation field lands on ImproveMethodResult (improve-types.ts:430), is computed at method-execution.ts:449-455, and is re-exported through improve.ts and index.ts. It is wired into the primary public improve() API that existing examples already call (examples/ablation-suite/gepa-driver-prompt.ts:203, examples/improve/improve.ts:97, `examples/intelligence-recomm
• Fit with existing patterns: Fits the established ownership boundary exactly: Eval owns the artifacts (readExternalOptimizerObservationArtifact, readGepaCandidatePopulationArtifact, decodeExternalTextCandidate from @tangle-network/agent-eval/campaign), Runtime owns materialization (reuses the same materializeProfile closure from method-execution.ts:371 and Interface's diffAgentProfiles/applyAgentProfileDiff/`c
• Real-world viability: Handles non-happy paths deliberately: methods reporting neither artifact get a clean unavailable status (method-execution.ts:103-108); callback-only candidates get explicit unavailable lineage (:219-222); materialization failures become structured refused candidates with error name/message (:226-237); the winner is cross-checked against the GEPA bestIndex candidate and against observ
• Model: opencode/zai-coding-plan/glm-5.2
• Bridge attempts: 1

No concerns — sound change, no better or existing approach found. ✅

---

What this audit checks

It judges the change on its merits — not whether it was tasked out in an issue. Unticketed, fast-moving work is fine; the question is whether the change is good and whether a better or existing approach should be used instead.

Pass	What it asks
Heuristic	Vague title? Whitespace-only or cruft-bearing diff? (content signals only)
Duplication	Do added function/class names already exist elsewhere in the repo?
Value Audit	What does it do? What goal does it achieve? Is it good? Better architecture or already-exists?
Usefulness Audit	Does it integrate and fit? Will it hold up in real use and actually get used?

Findings are concerns, not blocks — the human reviewer decides what to do with them.

_{value-audit · 20260811T044730Z}

[tangletools]

✅ No Blockers — 1f5ab447

Review health 100/100 · Reviewer score 57/100 · Confidence 95/100 · 19 findings (3 medium, 16 low)

	opencode GLM 5.2	opencode DeepSeek v4 Pro	opencode DeepSeek v4 Flash	aggregate
Readiness	80	70	57	57
Confidence	95	95	95	95
Correctness	80	70	57	57
Security	80	70	57	57
Testing	80	70	57	57
Architecture	80	70	57	57

Reviewer score is advisory once the run is complete and the verdict has no blockers.

Full multi-shot audit completed 8/8 planned shots over 20 changed files. Global verifier still owns final merge decision. | Full multi-shot audit completed 8/8 planned shots over 20 changed files. Global verifier still owns final merge decision. | Full multi-shot audit completed 8/8 planned shots over 20 changed files. Global verifier still owns final merge decision.

🟠 MEDIUM SkillOpt agent-eval-rpc version below package.json minimum — README.md

The SkillOpt install block (line 263) pins agent-eval-rpc==0.144.8, but the GEPA block was bumped to 0.144.11 (line 249) and package.json:173 requires >=0.144.11 <0.145.0. CHANGELOG 0.131.8 line 9 confirms: 'Consume Agent Eval 0.144.11 so Runtime can verify both optimizer population artifacts.' Running both pip install instructions as written would produce a version conflict. Running only the SkillOpt section would install a version below the runtime's m

🟠 MEDIUM proposalSequence contract contradicts its documented semantics — src/improvement/improve-types.ts

Doc comment says 'One-based position of the proposal in the verified observation artifact', but the value is the proposal line's GLOBAL sequence in the JSONL log. agent-eval openExternalOptimizerObservationLog increments one shared counter across proposal+evaluation+refusal lines (external-optimizer-observations.ts: expectedSequence = submittedCandidates+evaluations+refusals+1) and readExternalOptimizerObservationArtifact sets proposalSequence = observation.sequence, which the reader also enforces to equal the file line number (sequence === index+1). With interleaved evaluation lines a 2nd proposal reports sequence 3, not 2. Consumers aligning proposalSequence with proposal-indexed data silently misalign. Fix: rename/re-document as 'artifact line sequence' or compute a proposal counter.

🟠 MEDIUM GEPA candidate-graph read assumes the wrong storage — src/improvement/method-execution.ts

profileCandidatePopulation reads BOTH artifacts with optimizationRunOptions?.storage, but the two artifacts live in different stores. The observation log is written by agent-eval's GEPA method to input.runOptions.storage (consistent with this reader). The candidate graph, however, is written by the Python subprocess to the real-fs outputDir (${runDir}/external, mkdir via node:fs in gepa-optimization-method.ts:379-380) and is read by agent-eval itself with fsCampaignStorage() (gepa-optimization-method.ts:460-462, no storage passed). So for any caller that supplies a non-fs optimizationRunOptions.storage (the exact pattern used by this PR's own improve.test.ts populationMethod), agent-eval's method-level read succeeds but Runtime's readGepaCandidatePopulationArtifact({summary, storage: cus

🟡 LOW Changelog entry is accurate; only interpretive wording differs from code — CHANGELOG.md

Bullet 'Consume Agent Eval 0.144.11 so Runtime can verify both optimizer population artifacts' states a purpose that the code supports but does not literally encode: the peer range is >=0.144.11 <0.145.0 and the catalog pin is 0.144.11, both matching. The 'verify both artifacts' rationale is a reasonable reading of method-execution.ts's digest-keyed join but is editorial. Not blocking.

🟡 LOW SkillOpt block still pins stale agent-eval-rpc 0.144.8 — README.md

This PR bumps the shared bridgeInstall string in src/improvement/official-optimizers.ts:28 from 0.144.10 to 0.144.11, and that string is also emitted in SkillOpt install errors (official-optimizers.ts:33-34). The README SkillOpt block (line 263, unchanged in this diff) still tells users to install agent-eval-rpc==0.144.8. Pre-existing inconsistency, not introduced here, but it sits in the same file this shot reviewed and now diverges from both the GEPA block (0.144.11) and the code's error text. Impact is limited: per the documented semantics a SkillOpt run on the older rpc simply returns candidatePopulation status:'unavailable'. Fix: align the SkillOpt block to agent-eval-rpc==0

🟡 LOW SkillOpt install snippet pins agent-eval-rpc below the documented minimum — README.md

This diff bumped the GEPA install pin from 0.144.8 to 0.144.11 at line 249, but the SkillOpt install block at line 263 still reads agent-eval-rpc==0.144.8. docs/canonical-api.md:9 states agent-eval must satisfy >=0.144.11 <0.145.0, and src/improvement/official-optimizers.ts:28 hardcodes 0.144.11. A user following the SkillOpt snippet installs a version below the required floor. Bump line 263 to 0.144.11 for consistency. Documentation-only, no runtime impac

🟡 LOW Doc generator loses ReadonlyAgentProfile type for profile field — docs/api/index.md

Source declares profile: ReadonlyAgentProfile but docs render it as profile: object. ReadonlyAgentProfile is a documented type alias at line 10979. This is a systemic doc-generator issue (not unique to this PR).

🟡 LOW Doc generator loses anonymous type shape for selectionScores — docs/api/index.md

Source declares selectionScores: readonly { scenarioId: string; score: number }[] but docs render as readonly object[]. The GepaCandidateSelectionScore type exists in agent-eval/campaign but is undocumented (no TSDoc). Systemic doc-generator issue.

🟡 LOW Doc generator loses intersection member for gepaCandidateGraph — docs/api/index.md

Source declares ImprovementProfilePopulationArtifactSource & { bestIndex: number } but docs render the intersection member as & object. Systemic doc-generator issue.

🟡 LOW selectionScores renders without field breakdown in index.md — docs/api/index.md

selectionScores is documented as readonly object[] with no nested type declaration, so a reader cannot see the scenarioId: string; score: number fields that exist in src/improvement/improve-types.ts:254-257. Sibling property gepaCandidateGraph? gets a full 'Type Declaration' block (index.md:5088) because it is an intersection type. This is a typedoc rendering artifact of inline anonymous array element types, not a hand-authored error, and regeneration reproduces it deterministically. Impact is limited doc fidelity on an API whose docs brand exactness. No fix needed at PR level; note only.

🟡 LOW §2 decision table lacks a row for the new candidatePopulation surface — docs/canonical-api.md

The spine (step 4) introduces improve().candidatePopulation — a new public read surface — but §2 'I want to ___' table has no row telling a reader how to reach it (e.g. 'Inspect every verified optimizer candidate as exact profiles/diffs'). The pattern of this doc is intent→primitive mapping; a reader must now discover the field via generated docs. Not a correctness issue (field exists and is documented in docs/api), just a completeness nit. Fix: add a §2 row pairing the intent with improve(...).candidatePopulation.

🟡 LOW New candidate-source type is declared but not structurally exercised — scripts/verify-package-exports.mjs

ImprovementProfilePopulationCandidateSource is ambiently declared and voided but never narrowed or assigned from a real population candidate. This proves importability but not assignability from ImprovementProfileCandidatePopulationAvailable.candidates[].source. Consistent with other ambient declares in the file (e.g. profileStateResolver), so non-blocking; a future extension could derive it from candidatePopulation when status==='available' to tighten the surface guarantee.

🟡 LOW Inconsistent error classes for integrity failures — src/improvement/method-execution.ts

Artifact disagreement (line 142) and diff non-reproduction (line 243) throw plain Error while every adjacent integrity failure (winner mismatch, missing bestIndex node, executionRef format) throws ConfigError. improve()/runMethodImprovement otherwise present config and evidence failures uniformly as ConfigError. Harmless functionally, but callers that distinguish ConfigError to recover vs. hard-fail will treat these two integrity violations differently. Align on ConfigError for consistency.

🟡 LOW No test for observation-only population path (graph absent) — src/improvement/method-execution.ts

The else-if branch at L195-201 (winnerSurface must appear in entries when graph is absent) is not directly covered by a test. The existing populationMethod fixture provides both graph AND observations; no fixture provides observations without a graph. The code path is correct by inspection — it iterates entries and checks isDeepStrictEqual — but a dedicated test would harden this branch against future refactors.

🟡 LOW Redundant graph truthiness check in lineage status — src/improvement/method-execution.ts

The condition entry.lineageNodes.length > 0 && graph on line 210 includes a redundant && graph check. lineageNodes is only populated inside for (const graphCandidate of graph?.candidates ?? []) (line 168), so if graph is falsy, lineageNodes is guaranteed empty. The && graph can never be true when lineageNodes.length > 0 and graph is falsy. Harmless but misleading — a reader might infer graph could be undefined at that point.

🟡 LOW Three defensive error branches have no test coverage — src/improvement/method-execution.ts

Three thrown errors are never exercised by tests: (1) line 198-200 'method winner does not appear in the verified optimizer observations' (observation-only winner mismatch — only the graph mismatch at line 191 is tested); (2) line 243-245 'Interface profile diffs do not reproduce optimizer candidate' (diff round-trip failure); (3) [line 142](https://github.com/tangle-networ

🟡 LOW Unbounded artifact read; size bound checked only after full load — src/improvement/method-execution.ts

Both readGepaCandidatePopulationArtifact and readExternalOptimizerObservationArtifact call storage.read(path) first and verify size/digest only after the entire file is in memory; the observations artifact has no byte bound at all, and the graph bound (maximumArtifactBytes) is enforced post-read. The provenance summary comes from an optimizer subprocess that also writes the artifact bytes, so it can craft a matching digest with unbounded size. Runtime's new post-run read is a second unbounded load (agent-eval's method-level read already does one during optimize). Low exposure today because the same pattern predates this PR, but the new Runtime consumer widens the blast radius. Consider bounding bytes before load or rejecting oversized summaries.

🟡 LOW materializeProfile validation side-effects now fire for every population candidate — src/improvement/method-execution.ts

profileCandidatePopulation calls the wrapped materializeProfile (which runs methodControls.validateCandidate and user validateCandidate callbacks) for every unique candidate in the population, not just the winner. The validatedCandidates Set prevents duplicate validation per-digest, but user-supplied validateCandidate callbacks with side effects (logging, billing, state mutation) will now fire N times instead of 1. This is intentional and correct (all reported candidates should be validated), but it is an observable behavior change for callers with stateful validators. No fix needed; worth a one-line note in release notes.

🟡 LOW profileCandidatePopulation is a 188-line function with no internal decomposition — src/improvement/method-execution.ts

The function combines artifact reading, entry building, winner validation, candidate materialization, and diff computation in a single function body. Extracting the entry-construction loop, winner-verification guard, or the per-candidate materialize-and-diff step into named sub-functions would make the control flow more auditable without changing behavior.

---

_{tangletools · 2026-08-11T05:03:58Z · trace}

[tangletools]

✅ Approved — 19 non-blocking findings — 1f5ab447

Full multi-shot audit completed 8/8 planned shots over 20 changed files. Global verifier still owns final merge decision. | Full multi-shot audit completed 8/8 planned shots over 20 changed files. Global verifier still owns final merge decision. | Full multi-shot audit completed 8/8 planned shots over 20 changed files. Global verifier still owns final merge decision.

Full immutable report for this review: trace

Summary comment for this run: full summary

---

_{tangletools · 2026-08-11T05:03:58Z · immutable trace}