fix(routing): calibrate quality dial against real mix crossovers#479
Merged
Conversation
…ling dead zones The QualityBias dial (#476) bent a per-cluster alpha by quality dispersion and remapped it onto a cost band. That removed the *outer* flat regions but not interior ones: on the deployed bundle the routed mix is identical across alpha ~0.45-0.62, so the dial still showed the same model mix from ~20% all the way to ~55% ("50/50 looks exactly like 20/80"). The bend operates blind to where crossovers actually are, so it can't fix an interior dead zone. Replace it with a per-bundle calibration. At scorer build, sweep a uniform alpha 0->1, scoring every cluster centroid through the live blend, and record the alpha at which the routed mix (multiset of per-cluster winners) changes. Those breakpoints — ascending, first pinned to 0, last to 1 — are placed at equal dial spacing and interpolated, so equal dial travel crosses an equal number of mix changes. Every part of the slider now does something. On v0.68 the projected cost now rises smoothly $0.10 -> $3.88 per 1k input across the dial: t=0 is 100% cheapest, t=0.5 is gemini-3.1-pro-dominant, t=1 is the best-per-cluster mix (62% opus / 38% gemini-3.1-pro). The endpoints stay honest (all-cheapest / best-per-cluster); only the path between them is recalibrated. Removes the now-unused dispersion-rank + cost-band machinery. The Alpha sledgehammer (eval/debug header) path is unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Reviews (1): Last reviewed commit: "fix(routing): calibrate quality dial aga..." | Re-trigger Greptile |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
…zone test PR review (greptile): - computeDialCalibration now skips an empty argmax winner, matching RoutingDistribution's accounting, so a cluster flipping between "" and a real model can't inject a phantom dial breakpoint. - The mid-vs-low regression test now requires the 0.2 dial point to be present, so the assertion can't silently degrade to a no-op if the grid changes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This was referenced Jul 17, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Reported by a user testing the quality-vs-price dial: 50/50 routes essentially the same mix as 20/80 — both land on the two cheapest models, with nothing in between, and the mix only "wakes up" near the top of the slider.
The dispersion-bend dial from #476 bent a per-cluster alpha by quality dispersion and remapped it onto a cost band. That trimmed the outer flat regions (the all-cheapest floor and the saturated ceiling) but not interior ones. On the deployed bundle the routed mix is identical across alpha ≈ 0.45–0.62, so the dial still showed one frozen mix from ~20% to ~55%. The bend is computed blind to where crossovers actually sit, so it structurally can't remove an interior dead zone.
Fix
Calibrate the dial against the bundle's actual routing behavior. At scorer build, sweep a uniform alpha 0→1, score every cluster centroid through the same blend as live routing, and record the alpha at which the routed mix (the multiset of per-cluster winners) changes. Those breakpoints — ascending, first pinned to 0, last to 1 — are placed at equal dial spacing and interpolated by
dialToAlpha. Equal dial travel then crosses an equal number of mix changes, so every part of the slider moves the mix.This subsumes the dispersion bend (uniform alpha already routes high-dispersion clusters to quality earlier), so the per-cluster dispersion-rank + cost-band machinery is removed. The
Alphasledgehammer (eval/debug header) path is unchanged.Result (deployed v0.68, projected $/1k input)
Cost now rises smoothly and monotonically across the whole slider; the endpoints stay honest (all-cheapest / best-per-cluster).
Tests
dialToAlpha: endpoints pin, monotone, interpolates between breakpoints, identity fallback with no calibration.computeDialCalibration: ascending, endpoint-pinned, multiple interior breakpoints on the real bundle.RoutingDistribution: no dead zone (no long run of identical adjacent mixes), and the 50% dial routes a materially pricier mix than the 20% dial (the reported symptom, as a regression guard).🤖 Generated with Claude Code