chore(drift): classify claude-opus-5 in includeFamilies (live GA text chat)#339
Merged
Conversation
… chat) The live /models family canary is red on main: Anthropic shipped `claude-opus-5` (Claude Opus 5, created_at 2026-07-24) and the registry has no classification for it, so `unclassifiedFamilies` flags it as a new-family decision and keeps routing it to a human every run. Classify it as INCLUDE — it is a GA text-chat/reasoning family on the live listing, exactly like its already-included siblings `claude-sonnet-5` and `claude-opus-4-8`. All 11 ids the live Anthropic listing returns are text-chat Claude families and every other one is already in includeFamilies.anthropic; opus-5 was simply missed because it landed after the last classification wave (34056c2). No reasoning-map change. `isReasoningModel("claude-opus-5")` already returns true via the documented fail-open default (no NONREASONING_FAMILIES prefix matches), which is the correct verdict for an extended-thinking Opus-class model. That matches the precedent set for claude-sonnet-5 and claude-fable-5, which also rely on the default rather than a REASONING_FAMILIES entry, so src/model-utils.ts is untouched. No mock/fixture work needed: aimock's Anthropic surface is fixture-driven and model-agnostic, so claude-opus-5 already behaves identically to the included claude-sonnet-5. Re-pin includeFamilies.anthropic membership — the DATA_FROZEN canary correctly trips on the deliberate addition. New pin derived independently from the test's own sha256(JSON.stringify(sorted)) and cross-checked against vitest's reported value. Registry data is test/script-only (no shipped importer, absent from the npm tarball, dist/ semantically identical pre/post), so this carries zero behavior delta and no version bump.
commit: |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What was unclassified
claude-opus-5— Claude Opus 5 — is an unclassified Anthropic model family onmain. It appears in Anthropic's liveGET /v1/modelslisting but is in neitherincludeFamilies.anthropicnorexcludeFamilies.anthropic, so the live/modelsfamily canary flags it as a new-family decision and re-routes it to a human on every run.It was simply missed: the last classification wave (
34056c2) covered the 2026-07-16 listing, and Opus 5 landed after it (created_at: 2026-07-24).Only this one family is unclassified. The live listing returns 11 ids and every other one is already classified — including the other two Claude-5 members,
claude-sonnet-5andclaude-fable-5. So this is a one-family fix, not a wave.Evidence it was unclassified
Live listing (
GET https://api.anthropic.com/v1/models, fetched 2026-07-24, key sourced from the session secret cache — never persisted anywhere):Independently corroborated by PR #337's
drift-live-prjob, whose uploaded head and base drift reports both contain exactly one entry — this same finding — withconclusion: critical. It passed #337's delta gate only because that gate is new-in-head-only, so the finding has been sitting onmainunaddressed.Classification chosen, and why
includeFamilies.anthropic(src/__tests__/drift/model-registry.ts:132), under the existing// Claude 5 families (text chat)comment.Justification from the code, not from vibes:
includeFamiliesis documented as "families aimock actually MOCKS … chat / text completion families" (model-registry.ts:16-20). Opus 5 is a GA text-chat/reasoning model on the live listing — squarely that category.includeFamilies.anthropic.excludeFamilies.anthropicholds only retired/legacy ids (claude-v3,claude-2,claude-instant-1,model-registry.ts:217-222). Opus 5 is brand new, not retired, so exclude would be wrong.claude-sonnet-5andclaude-opus-4-8are bothincludeFamiliesentries added by34056c2("INCLUDE (text chat / reasoning aimock mocks): … Anthropic all 4.x/5 point releases"). Opus 5 is the same class of thing.PREVIEW_FAMILY,GEMMA_FAMILY) applies.Reasoning vs non-reasoning: reasoning-capable, and no code change needed
isReasoningModel("claude-opus-5")already returnstrue, before and after this PR. Verified empirically (not reasoned) by driving the realsrc/model-utils.tsexport on both tree states:trueis the correct verdict for an extended-thinking Opus-class model. It arrives via the documented fail-open default (model-utils.ts:18-26,135-136): noNONREASONING_FAMILIESprefix matchesclaude-opus-5, andREASONING_FAMILIES'claude-opus-4entry does not prefix-matchclaude-opus-5either, so it falls through totrue.So
src/model-utils.tsis deliberately untouched. Adding aREASONING_FAMILIESentry would be a no-op — that list is explicitly "mostly documentation; because unknown ids default totrueit is not load-bearing for correctness" (model-utils.ts:21-22) — and it would break the precedent thatclaude-sonnet-5andclaude-fable-5also rely on the default rather than an explicit entry. Touching shipped source for a zero-delta documentation edit would turn a registry-only change into a src change for no benefit.Mock / fixture support: none needed
Registry classification alone is sufficient, and this PR does not claim support aimock lacks.
aimock's Anthropic surface is fixture-driven and model-agnostic — it matches on request content, not on model id. Probed the built
dist/with real HTTP requests against a running server:Byte-identical behavior to the already-included
claude-sonnet-5: with a matching fixture loaded both serve it, without one both reportNo fixture matched. No per-family code path exists, so there is nothing family-specific to add.DEFAULT_MODELS(src/server.ts:227) is a separate static 5-id list that this registry does not feed and that no Claude-5 family was ever in.This is exactly the situation the existing
claude-fable-5comment already documents — a family included ahead of a recorded fixture, intentionally.RED → GREEN
The failure surface is the real live drift leg:
src/__tests__/drift/models.drift.ts:606-618,describe.skipIf(!process.env.ANTHROPIC_API_KEY), which callslistAnthropicModels()against the real API and asserts throughassertNoUnclassifiedFamilies. It gates onANTHROPIC_API_KEYand is SKIPPED without one, so reproducing the RED below requires credentials; a bare checkout silently skips it. A credential-free leg is included after.RED — unmodified
main(d1b5304), live APIGREEN — same command, after the classification
Credential-free corroboration
Driving the pure mirror
unclassifiedFamiliesForSync(scripts/drift-sync.ts) over the captured live payload, on both tree states:Membership pin RED → GREEN
The
DATA_FROZENcanary correctly tripped on the deliberate addition:The new pin was derived independently, not copied from vitest's
Received. Reimplemented the test's own hashing (sha256(JSON.stringify(members.sort())),logic-pin.test.ts:59-61+:227) in a standalone script, and first validated the method by reproducing the pre-existing pin exactly:Three-way agreement (derived / vitest
Received/ committed value). After re-pinning:Full suite — zero regressions
Versioning verdict: no bump, no changeset
No behavior change, so nothing to release. Three independent confirmations:
model-registry.tsis a test or a repo script (scripts/drift-sync.ts,scripts/drift-sync-check.ts, and sixsrc/__tests__/files). Non-test, non-script importers: none.src/model-utils.ts,src/server.tsand thetsdownentrypoints never touch it —includeFamilies/excludeFamiliesare drift metadata, not a runtime allowlist.package.jsonfilesis an allowlist (["dist","fixtures",".claude-plugin","skills","CHANGELOG.md"]);npm pack --dry-runyields 576 files with 0 matching__tests__/logic-pin/model-registry/src/.dist/contains zero occurrences ofincludeFamilies,excludeFamilies, orclaude-opus-5.dist/, normalizing the two known non-reproducible-build artifacts this repo has (tsdown's import-alias numbering, e.g.node_http0vsnode_http1, and declaration sourcemaps), with two identical-source builds as the noise floor:Release mechanism is a manual
package.jsonversion bump consumed bypublish-release.yml; there is no Changesets / release-please / semantic-release, and no workflow derives a version from commit messages.chore(drift):is therefore the honest prefix — this is a classification-only change with zero behavior delta, so neitherfeatnorfixis warranted.Pre-push gates
Pre-existing caveat (not introduced here)
.github/workflows/unreleased-check.ymlcounts commits touchingsrc/since the last tag without excludingsrc/__tests__/, so on merge it will emit::warning:: … version has not been bumped. It is a warning with noexit 1and does not fail CI. #336 and #337 have the same shape.Live-CI RED → GREEN (added after CI ran)
drift-live-prruns the drift suite against the real provider/modelslistings and uploads a base and a head report. On this PR (run 30132383648) those two artifacts are themselves the red-green, from CI rather than from my machine:mainis measurably red on this finding right now; this branch is clean. Full CI: 26 pass, 2 skipping, 0 fail (28 rows).