Skip to content

fix(routing): gate agentic pool to harness-capable models so the price dial can demote Opus#515

Merged
steventohme merged 1 commit into
mainfrom
steven/agentic-dial-pool-gate
Jun 25, 2026
Merged

fix(routing): gate agentic pool to harness-capable models so the price dial can demote Opus#515
steventohme merged 1 commit into
mainfrom
steven/agentic-dial-pool-gate

Conversation

@steventohme

Copy link
Copy Markdown
Collaborator

Problem

The QualityBias price/quality dial could not bias cheaper for the agentic/coding clusters — Opus 4.8 stayed in rotation at every dial position.

Root cause: the guard was on the wrong axis. applyDialAlpha floors the per-cluster quality weight (alpha[i] = max(dialAlpha, floor[i])) at 0.88 for agentic clusters. A single scalar weight can't express "cheaper but still harness-capable": the agentic quality band is compressed while cost spans the full deployed range, so alpha has only two regimes — high → Opus pinned, low → cheapest-incapable model wins (the original minimax-m3 "grepped for a skill instead of running it" cliff). 0.88 (from #501) avoided the cliff by disabling the dial for agentic entirely.

Fix — move the guard from the weight to the candidate pool

  1. Catalog: new AgenticUse capability axis (AgenticLow) — stricter than ToolUseLow: a model emits valid tool calls but can't sustain the skill/tool orchestration loop. Flags minimax-m3, gemini-3.1-flash-lite, deepseek-v4-flash, qwen3-next-80b-instruct. AgenticLowSet() mirrors ToolUseLowSet(). (mimo-pro left eligible — the catalog's documented sweep says the pro variant is stable.)
  2. Scorer: on has_tools turns, subtract AgenticLowSet from the eligible pool via the existing soft-filter seam (empty-pool fallback preserved, so a decision is always returned).
  3. v0.70 metadata: relax agentic alpha_floor 0.88 → 0.30 (in place, matching fix(routing): per-cluster alpha_floor so the price/quality dial keeps every cluster at its best-for-budget model #501's pattern) so the dial walks the full harness-capable ladder.

Validated behavior (probed the real v0.70 bundle)

Agentic demotion ladder is now:

dial model
quality end claude-opus-4-8
mid gemini-3.1-pro-preview
cost end gpt-5.4-mini

The gate is load-bearing at the cost end: without it the pool falls through to gemini-3.1-flash-lite (incapable); with it the cheapest capable coder wins. Covered by TestScorer_HasToolsExcludesAgenticLowFromArgmax, the rewritten TestApplyDialAlpha_AgenticStaysOnCapableModelAtLowDial, and catalog set tests.

Known follow-up

The dial preview (RoutingDistribution) has no has_tools signal, so it scores over the full pool and will under-represent the gate at low dial (shows gemini-flash-lite for agentic clusters where real tool turns get gpt-5.4-mini). Actual routing is correct. Fully unifying them means making the gate cluster-level in blendScoresV2.

Notes

  • v0.70 geometry (centroids / quality_means) is unchanged — only routing knobs + a pool gate. Deploys via the normal gitlink bump.
  • Offline sign-off: cmd/routing-report --quality-bias.

🤖 Generated with Claude Code

…e dial can demote Opus

The QualityBias dial could not bias cheaper for agentic/coding clusters: the
per-cluster alpha_floor (0.88) clamped the quality WEIGHT so high that Opus won
every dial position. The floor was the wrong lever — a single scalar weight
can't express "cheaper but still harness-capable" because the agentic quality
band is compressed while cost spans the full range, so any low-enough alpha
falls straight off the cliff to the cheapest model (the original minimax-m3
"grepped for a skill instead of running it" failure). 0.88 "fixed" that by
disabling the dial for agentic entirely.

Move the guard from the weight to the candidate POOL:

- catalog: new AgenticUse capability axis (AgenticLow), stricter than
  ToolUseLow — emits valid tool calls but can't drive the skill/tool harness.
  Flags minimax-m3, gemini-3.1-flash-lite, deepseek-v4-flash,
  qwen3-next-80b-instruct. AgenticLowSet() mirrors ToolUseLowSet().
- scorer: on has_tools turns, subtract AgenticLowSet from the eligible pool via
  the existing soft-filter seam (empty-pool fallback preserved).
- v0.70 metadata: relax agentic alpha_floor 0.88 -> 0.30 so the dial walks the
  full harness-capable ladder.

Result on the v0.70 bundle: agentic demotion ladder is now Opus (quality end)
-> gemini-3.1-pro (mid) -> gpt-5.4-mini (cost end). The gate is load-bearing at
the cost end: without it the pool falls through to gemini-3.1-flash-lite (an
incapable model); with it the cheapest capable coder wins.

Known follow-up: the dial preview (RoutingDistribution) has no has_tools signal
and scores over the full pool, so it under-represents the gate at low dial.
Unifying it means making the gate cluster-level in blendScoresV2.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown

🧭 Cluster routing review

ℹ️ Informational — candidate artifact (latest unchanged); no sign-off required.

Cluster routing report — v0.70

Routed 180 labeled probes through the real Scorer at top_p=2, embedder jina-v2-base-code-int8. Premium = opus-4-7/4-8, gpt-5.5, gemini-3.1-pro, sonnet-4-6.

Register → model (this artifact)

register Opus % premium % top models
conversational 33% 43% deepseek-v4-pro:16 claude-opus-4-8:10 gemini-3.1-pro-preview:2 gemini-3.5-flash:1
trivial_nl 13% 50% deepseek-v4-pro:13 gemini-3.1-pro-preview:10 claude-opus-4-8:4 gpt-5.5:1
knowledge_qa 70% 77% claude-opus-4-8:21 glm-5.2:4 deepseek-v4-pro:2 gemini-3.1-pro-preview:1
easy_code 73% 87% claude-opus-4-8:22 deepseek-v4-pro:3 gemini-3.1-pro-preview:2 gpt-5.5:2
hard_code 60% 80% claude-opus-4-8:18 gpt-5.5:6 glm-5.2:6
agentic_tool 87% 93% claude-opus-4-8:26 gpt-5.5:2 glm-5.2:2

What each cluster represents (probe register mix by nearest centroid)

cluster probes dominant register breakdown
0 36 agentic_tool agentic_tool:12 knowledge_qa:9 hard_code:8 conversational:4 easy_code:2 trivial_nl:1
1 8 easy_code easy_code:3 agentic_tool:2 conversational:1 knowledge_qa:1 trivial_nl:1
2 2 conversational conversational:2
3 1 knowledge_qa knowledge_qa:1
4 3 conversational conversational:1 easy_code:1 trivial_nl:1
5 14 agentic_tool agentic_tool:5 conversational:2 easy_code:2 hard_code:2 knowledge_qa:2 trivial_nl:1
6 1 easy_code easy_code:1
7 6 agentic_tool agentic_tool:2 easy_code:2 hard_code:1 knowledge_qa:1
8 9 knowledge_qa knowledge_qa:4 agentic_tool:2 hard_code:2 easy_code:1
9 10 conversational conversational:4 trivial_nl:4 easy_code:1 knowledge_qa:1
10 24 conversational conversational:12 trivial_nl:7 easy_code:3 knowledge_qa:2
11 4 trivial_nl trivial_nl:2 easy_code:1 knowledge_qa:1
12 13 easy_code easy_code:9 knowledge_qa:2 agentic_tool:1 trivial_nl:1
13 25 hard_code hard_code:17 agentic_tool:6 easy_code:1 knowledge_qa:1
14 7 knowledge_qa knowledge_qa:4 easy_code:2 trivial_nl:1
15 17 trivial_nl trivial_nl:11 conversational:4 easy_code:1 knowledge_qa:1

@greptile-apps

greptile-apps Bot commented Jun 25, 2026

Copy link
Copy Markdown

T-Rex T-Rex Logs

What T-Rex did

  • T-Rex attempted the Go test for TestTrexAgenticLadderV070HasTools, first running the base command and then the head-target command; both runs failed with exit code 127 and go: command not found.
  • The environment blocker was identified: the PATH does not contain a go binary and network installation was disallowed, preventing Go from being installed.

View all artifacts

T-Rex Ran code and verified through T-Rex

Reviews (1): Last reviewed commit: "fix(routing): gate agentic pool to harne..." | Re-trigger Greptile

@steventohme
steventohme merged commit da9a1bf into main Jun 25, 2026
9 checks passed
@steventohme
steventohme deleted the steven/agentic-dial-pool-gate branch June 25, 2026 18:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant