fix(routing): gate agentic pool to harness-capable models so the price dial can demote Opus#515
Merged
Merged
Conversation
…e dial can demote Opus The QualityBias dial could not bias cheaper for agentic/coding clusters: the per-cluster alpha_floor (0.88) clamped the quality WEIGHT so high that Opus won every dial position. The floor was the wrong lever — a single scalar weight can't express "cheaper but still harness-capable" because the agentic quality band is compressed while cost spans the full range, so any low-enough alpha falls straight off the cliff to the cheapest model (the original minimax-m3 "grepped for a skill instead of running it" failure). 0.88 "fixed" that by disabling the dial for agentic entirely. Move the guard from the weight to the candidate POOL: - catalog: new AgenticUse capability axis (AgenticLow), stricter than ToolUseLow — emits valid tool calls but can't drive the skill/tool harness. Flags minimax-m3, gemini-3.1-flash-lite, deepseek-v4-flash, qwen3-next-80b-instruct. AgenticLowSet() mirrors ToolUseLowSet(). - scorer: on has_tools turns, subtract AgenticLowSet from the eligible pool via the existing soft-filter seam (empty-pool fallback preserved). - v0.70 metadata: relax agentic alpha_floor 0.88 -> 0.30 so the dial walks the full harness-capable ladder. Result on the v0.70 bundle: agentic demotion ladder is now Opus (quality end) -> gemini-3.1-pro (mid) -> gpt-5.4-mini (cost end). The gate is load-bearing at the cost end: without it the pool falls through to gemini-3.1-flash-lite (an incapable model); with it the cheapest capable coder wins. Known follow-up: the dial preview (RoutingDistribution) has no has_tools signal and scores over the full pool, so it under-represents the gate at low dial. Unifying it means making the gate cluster-level in blendScoresV2. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
🧭 Cluster routing review
Cluster routing report —
|
| register | Opus % | premium % | top models |
|---|---|---|---|
| conversational | 33% | 43% | deepseek-v4-pro:16 claude-opus-4-8:10 gemini-3.1-pro-preview:2 gemini-3.5-flash:1 |
| trivial_nl | 13% | 50% | deepseek-v4-pro:13 gemini-3.1-pro-preview:10 claude-opus-4-8:4 gpt-5.5:1 |
| knowledge_qa | 70% | 77% | claude-opus-4-8:21 glm-5.2:4 deepseek-v4-pro:2 gemini-3.1-pro-preview:1 |
| easy_code | 73% | 87% | claude-opus-4-8:22 deepseek-v4-pro:3 gemini-3.1-pro-preview:2 gpt-5.5:2 |
| hard_code | 60% | 80% | claude-opus-4-8:18 gpt-5.5:6 glm-5.2:6 |
| agentic_tool | 87% | 93% | claude-opus-4-8:26 gpt-5.5:2 glm-5.2:2 |
What each cluster represents (probe register mix by nearest centroid)
| cluster | probes | dominant | register breakdown |
|---|---|---|---|
| 0 | 36 | agentic_tool | agentic_tool:12 knowledge_qa:9 hard_code:8 conversational:4 easy_code:2 trivial_nl:1 |
| 1 | 8 | easy_code | easy_code:3 agentic_tool:2 conversational:1 knowledge_qa:1 trivial_nl:1 |
| 2 | 2 | conversational | conversational:2 |
| 3 | 1 | knowledge_qa | knowledge_qa:1 |
| 4 | 3 | conversational | conversational:1 easy_code:1 trivial_nl:1 |
| 5 | 14 | agentic_tool | agentic_tool:5 conversational:2 easy_code:2 hard_code:2 knowledge_qa:2 trivial_nl:1 |
| 6 | 1 | easy_code | easy_code:1 |
| 7 | 6 | agentic_tool | agentic_tool:2 easy_code:2 hard_code:1 knowledge_qa:1 |
| 8 | 9 | knowledge_qa | knowledge_qa:4 agentic_tool:2 hard_code:2 easy_code:1 |
| 9 | 10 | conversational | conversational:4 trivial_nl:4 easy_code:1 knowledge_qa:1 |
| 10 | 24 | conversational | conversational:12 trivial_nl:7 easy_code:3 knowledge_qa:2 |
| 11 | 4 | trivial_nl | trivial_nl:2 easy_code:1 knowledge_qa:1 |
| 12 | 13 | easy_code | easy_code:9 knowledge_qa:2 agentic_tool:1 trivial_nl:1 |
| 13 | 25 | hard_code | hard_code:17 agentic_tool:6 easy_code:1 knowledge_qa:1 |
| 14 | 7 | knowledge_qa | knowledge_qa:4 easy_code:2 trivial_nl:1 |
| 15 | 17 | trivial_nl | trivial_nl:11 conversational:4 easy_code:1 knowledge_qa:1 |
What T-Rex did
Reviews (1): Last reviewed commit: "fix(routing): gate agentic pool to harne..." | Re-trigger Greptile |
This was referenced Jul 17, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The QualityBias price/quality dial could not bias cheaper for the agentic/coding clusters — Opus 4.8 stayed in rotation at every dial position.
Root cause: the guard was on the wrong axis.
applyDialAlphafloors the per-cluster quality weight (alpha[i] = max(dialAlpha, floor[i])) at0.88for agentic clusters. A single scalar weight can't express "cheaper but still harness-capable": the agentic quality band is compressed while cost spans the full deployed range, so alpha has only two regimes — high → Opus pinned, low → cheapest-incapable model wins (the originalminimax-m3"grepped for a skill instead of running it" cliff).0.88(from #501) avoided the cliff by disabling the dial for agentic entirely.Fix — move the guard from the weight to the candidate pool
AgenticUsecapability axis (AgenticLow) — stricter thanToolUseLow: a model emits valid tool calls but can't sustain the skill/tool orchestration loop. Flagsminimax-m3,gemini-3.1-flash-lite,deepseek-v4-flash,qwen3-next-80b-instruct.AgenticLowSet()mirrorsToolUseLowSet(). (mimo-proleft eligible — the catalog's documented sweep says the pro variant is stable.)has_toolsturns, subtractAgenticLowSetfrom the eligible pool via the existing soft-filter seam (empty-pool fallback preserved, so a decision is always returned).alpha_floor0.88 → 0.30(in place, matching fix(routing): per-cluster alpha_floor so the price/quality dial keeps every cluster at its best-for-budget model #501's pattern) so the dial walks the full harness-capable ladder.Validated behavior (probed the real v0.70 bundle)
Agentic demotion ladder is now:
claude-opus-4-8gemini-3.1-pro-previewgpt-5.4-miniThe gate is load-bearing at the cost end: without it the pool falls through to
gemini-3.1-flash-lite(incapable); with it the cheapest capable coder wins. Covered byTestScorer_HasToolsExcludesAgenticLowFromArgmax, the rewrittenTestApplyDialAlpha_AgenticStaysOnCapableModelAtLowDial, and catalog set tests.Known follow-up
The dial preview (
RoutingDistribution) has nohas_toolssignal, so it scores over the full pool and will under-represent the gate at low dial (showsgemini-flash-litefor agentic clusters where real tool turns getgpt-5.4-mini). Actual routing is correct. Fully unifying them means making the gate cluster-level inblendScoresV2.Notes
cmd/routing-report --quality-bias.🤖 Generated with Claude Code