feat(#975): model roles — semantic indirection between phase policy and concrete model strings - #979
Merged
admarble merged 4 commits intoAug 30, 2026
Conversation
…nd model strings Adds `run.modelRoles` (shipped defaults: fast=sonnet, strong=opus, frontier=fable) so config expresses roles; a single resolver maps roles to concrete model strings. Phase policy, --models, and ladder entries use `role:<name>` to reference an entry; bare strings pass through verbatim for full backward compat (AC-3). - `resolveRoleToModel` exported from config-resolver.ts; called inside `resolvePhasePolicies` so both ExecutionConfig producers resolve roles through the same function with no second implementation (AC-5). - A missing role fails loudly at config-resolution time before any agent session spawns, naming the role and available map keys (AC-2). - `agents.model` now accepts a free string (not closed enum) — accepts dated IDs and new family aliases (AC-6). - `requestedModel` / `resolvedModel` added to per-phase phasePolicies in metrics schema and PhaseMarker — enables cross-time benchmark comparisons as the roster moves under aliases (AC-4). - 34 new tests covering AC-1 through AC-6 including a source-inspection drift guard for the two-producer invariant (AC-5). Entire-Checkpoint: 3c96ca19bf4a
6 tasks
- Adds enrichPhasePoliciesFromResults to run-orchestrator, merging the concrete modelUsage key from driver results into phasePolicies before metrics write (AC-4 resolvedModel) - Adds requestedModel capture in resolvePhasePolicies for role: prefixes (AC-4 pre-resolution value) - Adds run-orchestrator.model-roles.test.ts with 8 tests covering AC-4 - Expands config-resolver.model-roles.test.ts with AC-4 requestedModel assertions and AC-5 producer-2 drift guard for ready.ts - Updates settings-schema.test.ts for AC-6: agents.model is now a free string (z.string()), accepting any alias or dated ID - Adds modelUsage field to AgentResult in agent-driver.ts - Threads resolvedModel from phase-executor through the PhaseResult type Entire-Checkpoint: c018f9472e8c
The first QA pass found AC-4's fields declared but never populated. The enrichment helper fixes that, but its unit tests cannot catch the call being dropped from the metrics record — pin the call site with a source-inspection guard, same pattern as the AC-5 producer-2 test. Claude-Session: https://claude.ai/code/session_0178TZyoqDoCYehwfnQUDyNW
…ugh, phase-executor resolvedModel extraction, PhaseMarkerSchema round-trip Entire-Checkpoint: 99aaa03d41f6
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
run.modelRolesmap with shipped defaults (fast=sonnet,strong=opus,frontier=fable) so phase policy,--models, and ladder entries can reference roles with arole:<name>prefix instead of hard-coding model stringsresolveRoleToModelis the single choke point — called insideresolvePhasePoliciesso both ExecutionConfig producers resolve roles through the same function (no drift, AC-5)agents.modelnow accepts any free string instead of the closedhaiku|sonnet|opusenum (AC-6)requestedModel/resolvedModelfields to per-phase metrics andPhaseMarkerfor cross-time benchmark comparisons (AC-4)AC Verification
run.modelRolesparses;role:<name>dispatches with mapped modelconfig-resolver.ts:resolveRoleToModel,resolvePhasePolicieswiring; 10 testsresolveRoleToModelthrows before any session; 4 testsrole:prefix → early return; existing tests green + 3 new testsrequestedModel+resolvedModelmetrics-schema.ts,state-schema.tsupdated; 4 metrics testsbuildExecutionConfig+commands/ready.tsboth callresolvePhasePolicies; source-inspection drift guard (3 tests)agents.modelfree string; docs no longer rank capabilityz.string().default("haiku"); JSDoc updated; 4 testsKey Decisions
role:prefix is the dispatch signal — bare strings always pass through (AC-3 backward compat){ "claude-code": value }only (per spike(config): model-role shape — flat map vs per-driver map, role vocabulary (gates the modelRoles feature) #974); cross-driver form uses object with driver keyssequant setupwrites nothing formodelRoles— absent key → shipped defaults applyTest Plan
npm run build— passes (0 TypeScript errors)npm run lint— passes (0 ESLint errors)config-resolver.phase-policy.test.tstests still pass (no regression)Closes #975
🤖 Generated with Claude Code
Orchestrator verification (2026-08-29, commits d229de8 + 8bfb8fe)
Independently verified in the worktree after the AC-4 fix round: all model-roles suites pass (config-resolver 37 incl. AC-5 producer-2 drift guard; run-orchestrator AC-4 9/9; settings/metrics-schema), build + lint clean. Data-flow traced end to end: claude-code driver captures
modelUsage→AgentPhaseResult.modelUsage→ phase-executor extracts the first key asresolvedModel(phase-executor.ts:1381-1386) →enrichPhasePoliciesFromResultsmerges it per phase → the metrics record'sphasePoliciesat run-orchestrator.ts:1778.requestedModelis captured at resolution time forrole:references.Added in 8bfb8fe: a source-inspection guard pinning the metrics call site — the first QA pass's defect (fields declared, never populated) could otherwise recur by dropping the call while every helper unit test stays green.
Mutation-verified: AC-4 — replaced the metrics record's
enrichPhasePoliciesFromResults(...)call with a plainconfig.phasePoliciespass-through; test "run-orchestrator's metrics record passes phasePolicies through enrichPhasePoliciesFromResults" failed; restored.Honest divergence flagged for the owner — AC-4's phase-marker clause
AC-4 says run metrics and the phase marker record requested + resolved model. The metrics half is fully implemented and gated. The marker half is schema-only (
PhaseMarkerSchemacarries the optional fields) and — as written — architecturally misfit:SEQUANT_PHASEmarkers are emitted by the phase agents themselves from the skill templates, before the driver'smodelUsageexists, so an agent cannot put the resolved concrete ID in its own marker (formatPhaseMarkerhas no production caller; markers come from skill-templated comments). Options: (a) accept metrics as AC-4's system of record (the #916/#944 consumers read metrics, not markers) and drop the marker clause; (b) a small follow-up threadingrequestedModelinto the marker via the existingSEQUANT_*env contract (feasible) whileresolvedModelstays metrics-only. Not silently rationalized — owner's call at review.