fix: derive thinking-strength controls from models.dev reasoning_options - #2050
Merged
Conversation
models.dev declares per-model reasoning_options, but the sync script only read the reasoning boolean, so generated metadata could never carry effort options. Thinking options were hand-declared per provider path in STATIC_MODEL_METADATA and the wire was a second hand-maintained switch; both silently missed most providers (42 of 44 had gaps), so reasoning models showed no thinking knob and, even when forced, sent nothing. - sync-model-metadata.mjs passes reasoning_options through (effort + toggle; budget_tokens recognized, no wire consumer yet) and is now importable with a main guard so the transform is unit-testable - buildProviderOptions falls back to a wire family mapper keyed on the resolved adapter kind (the same object getAIModel switches on, including per-model models.dev package overrides), so every upstream-declared variant has a wire; namespace derivation shared with getAIModel so the two cannot drift - github-copilot and xai non-grok-4.5 paths wired; cohere north-mini suppresses the undeclared effort levels its wire cannot express - contract test: every declared thinking variant on every provider path must wire non-empty (closes the silent 'forgot' state) - snapshot regenerated (models.dev drift included: 80 removed, 149 added, 668 existing models gain thinkingOptions); opencode-free pin updated for the deprecated hy3-free; vision tests re-pinned to models still present Fixes #1858
…allback The switch enumerated the same reasoningEffort one-liner for eleven provider paths (vercel, ollama-cloud, deepinfra, openrouter, groq, deepseek, moonshot, tencent-token-plan, zai-coding-plan, stepfun-step- plan, stepfun-ai-step-plan). All are now handled by the resolved-adapter family fallback with identical reachable behavior: the shape-B providers never declare an off level (verified against the snapshot), so the off->none mapping change is unreachable there. Also deleted the identity openaiCompatibleNamespace helper (every mapping was the provider type itself) in favor of the shared openAiCompatibleProviderOptionsName rule.
… contract, redundant pins Review feedback (three independent deepseek-v4-flash reviews) surfaced one real wire bug and several hardening items: - cohere north-mini-code-1-0 carries a models.dev package override to the OpenAI-compatible endpoint, but the explicit cohere case wired it with the native Cohere thinking object. Moved the cohere wire into the family mapper keyed on the resolved adapter kind (native cohere models keep thinking.disabled; overridden models get reasoning_effort) and dropped the static suppression that hid its declared efforts. - groq qwen3-32b: models.dev declares ['none','default'], which is qwen3.6-27b's value set misapplied; Groq docs and the repo's own verified comment say qwen3-32b has no reasoning_effort. Pinned it to no options until a live check proves otherwise. - contract test now sweeps the generated model universe (not just fallbackModels), requires a non-empty inner wire shape, and requires a real off wire for declared off levels. - removed 22 static thinkingOptions pins that were byte-identical to the regenerated snapshot (they were pure duplication created by the passthrough); kept pins that are the only source or carry real deltas. - displayMetadataOnly now inherits source thinkingOptions when no override exists, so claude-subscription automatically inherits upstream anthropic declarations instead of requiring hand-pins. - added sync tests for budget_tokens, unknown option types, and toggle-only models. - xai non-grok-4.5 path delegates to the shared family wire instead of duplicating it.
…rseded guards Second review round (gpt-5.6-sol, k3-256k) found the contract sweep's enumeration was a hand-mirror of lookupModelMetadata's alias rules plus fallback models only, so alias drift or static-only models (e.g. the claude-subscription derived table) could silently shrink coverage. - export modelMetadataIdsForProvider from core: the metadata universe per provider (generated snapshot + static overrides) under the same alias rules lookupModelMetadata applies; the sweep and the lookup can no longer drift. Removes the need for the model-metadata.generated subpath export added in the previous round. - drop the 19-case resolver/options drift guard from model-factory-thinking.test.ts: it asserted the same invariant as the contract sweep with a weaker JSON-substring off-wire check, and every case is now covered by the sweep's stricter exact-value checks. - delete empty static records left after pin removal (deepseek, stepfun-ai-step-plan, xai) and the now-duplicate tencent-token-plan hy3 pin (hy3's effort set comes from the snapshot; hy3-preview stays). - add a github-copilot protocol-branch test (anthropic-messages and openai-responses wires) that the sweep cannot reach without account models.
Astro-Han
marked this pull request as ready for review
August 3, 2026 19:30
Astro-Han
added a commit
that referenced
this pull request
Aug 4, 2026
…iniMax coding plans (#2067) * fix: make Kimi Coding Plan thinking strength follow the chosen level Kimi Coding Plan was a pre-#2050 orphan: it was missing from the sync-model-metadata PROVIDERS map, so its metadata stayed a hand-written static block (k3 pinned to a single 'max' effort, k3-256k absent), and the runtime wire hard-coded max and never read the level. - sync-model-metadata.mjs maps kimi-coding-plan to models.dev's kimi-for-coding provider; the snapshot now declares k3 (toggle + low/high/max) and k3-256k (low/high/max), and the stale static block is deleted (regenerated snapshot also carries a little openrouter drift from upstream) - buildProviderOptions passes the chosen level through both Kimi wires (anthropic effort and openai-chat reasoningEffort), defaulting to max when unset; k3-256k joins the K3 adaptive-thinking wire - the thinking wire contract test now sweeps kimi-coding-plan's declared levels automatically, closing the silent-forget gap for this provider Live-check pending: confirm Kimi's coding API accepts low/high for k3/k3-256k before relying on non-max levels. * fix: wire StepFun Step Plan metadata to its own models.dev provider stepfun-step-plan was a dormant twin of the kimi-coding-plan orphan: the sync map only covered stepfun / stepfun-ai / stepfun-ai-step-plan, so stepfun-step-plan lived on a hand-written static block that expanded stepfun (API) records and pinned reasoning facts by hand. - sync-model-metadata.mjs maps stepfun-step-plan to its own models.dev provider; the snapshot now declares all four Step Plan models with upstream effort levels (step-3.7-flash low/medium/high, the 3.5-flash pair low/high) and the static block is deleted - provider-registry.ts: stepfun-step-plan modelsDevId was mislabeled as stepfun; now resolves its own provider facts and the model guard checks the stepfun-step-plan snapshot segment - note: step-router-v1's reasoning capability now follows models.dev (false) instead of the hand-written true; no thinking options were declared either way, so the wire behavior is unchanged Declared levels are swept by the existing thinking wire contract, so this closes the same silent-forget gap for stepfun-step-plan. * fix(core): point MiniMax Coding Plan at its own models.dev segment and guard the sync map The MiniMax Coding Plan access path was the last alias-style provider: it declared modelsDevId minimax and reached metadata through a static alias to the MiniMax API segment, so the two models.dev segments could drift apart without anyone noticing (their docsUrl already differs). - sync-model-metadata.mjs maps minimax-coding-plan to its own models.dev provider; the snapshot now carries a dedicated segment (7 models, MiniMax-M3 toggle preserved) and the alias line is deleted - provider-registry.ts: minimax-coding-plan modelsDevId resolves its own segment facts and the fallback model guard checks its own segment - new models-dev-sync-contract test closes the silent-forget gap for good: every registry modelsDevId must resolve to the provider's own snapshot segment (or alias metadata), and every snapshot segment must be declared by the registry with its own id — adding a provider to the sync map without wiring its registry entry is now a test failure - registry modelsDevId now declared for every snapshot-backed provider: anthropic, openai, google, deepseek, kimi-coding-plan, zai-coding-plan, MiniMax, MiniMax-cn, gemini-cli (aliases xai-oauth/opencode-free keep their existing alias ids) * fix(core): close the silent-forget gap with a three-way models.dev sync contract Independent review of the previous fix found the sync contract only covered the already-declared universe: a registry provider with no modelsDevId and no sync-map entry (the exact kimi-coding-plan bug shape) was invisible to both directions, and the forward check was tautological (every modelsDevId was assigned from the same generated facts object it compared against). - sync-model-metadata.mjs now also emits GENERATED_MODELS_DEV_DIRECTORY: the complete models.dev provider catalog (id, name, api), not just the mapped subset - models-dev-sync-contract.test.ts becomes three-way: * forward: a declared modelsDevId must name a provider that exists in the directory (catches sync keys pointing at nonexistent sources) * reverse: every snapshot segment must be declared by the registry (unchanged) * directory: a registry provider whose base URL host matches a directory provider must declare modelsDevId — the kimi bug shape is now a test failure. Localhost/self-hosted endpoints are excluded - registry: kimi-coding-plan fallback models now guarded at import time like stepfun/minimax; claude-subscription and openai-codex declare their anthropic/openai segment sources; kimi-for-coding comment notes the intentionally-missing knob is not a sync gap - runtime: kimi case explicitly rejects off (future-proofing for a models.dev 'none' declaration) instead of the dead level !== 'off' guard; openai-chat namespace comment documents the ai-sdk camelCase alias dependency; highspeed branch comment states the no-wire intent - tests: openai-chat high passthrough asserted, kimi off entry-gate behavior asserted, step-router-v1 no-variants intent locked Verified the new contract has teeth: removing kimi's modelsDevId makes both the reverse and directory checks fail. * fix: address Claude Opus and Codex review — off rejection, alias contract, main() coverage Two independent external reviews (Claude Opus, Codex) both found no P0/P1, but converged on real gaps in the previous review round: - kimi 'off' rejection was dead code: the entry gate normalized off to undefined before the case guard, so an explicit off silently became max (the test even locked that in). The case now checks the raw thinkingLevel argument and rejects off with empty options. - the sync contract could not see alias-type orphans: a provider declaring a *neighbour* segment id (the exact pre-fix stepfun/minimax shape) passed all three checks, and claude-subscription pointing at openai passed too. Replaced host-based matching with a deterministic completeness check: every provider must have its own segment, a declared alias (xai-oauth/opencode-free/claude-subscription/openai- codex), or a whitelist reason (9 local/user-configured/volcengine entries). Both mutation shapes now fail the suite. - sync-model-metadata.mjs main() had zero coverage (the place the orphan bug lived): added an end-to-end fixture test over the real main() path, and exported PROVIDERS/main(argv) to make it testable. - the wire-contract sweep only exercised kimi's anthropic branch: added an explicit openai-chat sweep for kimi-coding-plan. - GENERATED_MODELS_DEV_DIRECTORY shrank to { api?: string } (name was never consumed); dropped the unused FACTS_BY_PROVIDER alias. - stepfun guard now covers all 4 fallback models (slice(0,3) was stale); minimax guard label says MiniMax Coding Plan. Deferred (recorded, not implemented): k3 low/high live check against the Kimi API (B1), kimi protocol-decision consolidation onto resolveModelRuntime (B3) and camelCase namespace cleanup (D3) as a runtime refactor, kimi-for-coding forced-max semantics pending the live check, user-visible upstream drift noted in the PR description. Verified: core 772/772, runtime 3089 pass / 4 pre-existing macOS symlink failures, sync scripts 9/9, lint/format/typecheck clean. Mutation tests: claude-subscription→openai and minimax-coding-plan→MiniMax.id both now fail the contract suite. * style: fix biome format in the kimi openai-chat sweep test
This was referenced Aug 20, 2026
Astro-Han
added a commit
that referenced
this pull request
Aug 20, 2026
The committed models.dev snapshot had drifted far enough that it no longer described the catalog it claims to mirror: 1690 models to 1779, deprecations 82 to 98, and pricing rows 1239 to 1316 (+86 new, -9 removed, 37 repriced). The refresh is regenerated from one pinned api.json payload rather than a live fetch, so every number here is reproducible from a fixed input. Three consumers had to follow. - Upstream deprecated two of the four ids pinned for the OpenCode Free offering, `north-mini-code-free` and `laguna-s-2.1-free`. Both are removed. `provider-registry.ts` asserts its pins are active and tool-capable at module load, so leaving them would fail the process at import, not at first use. - `gemini-3-pro-preview` left the catalog, so the Google thinking test moves to `gemini-3.1-pro-preview`, whose effort values carry the same no-`off` shape the test asserts. Two Google fallback ids the snapshot no longer contains, `gemini-2.0-flash` and `gemini-1.5-pro`, are dropped from that list. - The snapshot now sits about 2 KB under Biome's 1 MiB default, and this refresh alone added 57 KB, so the next one crosses it. An over-size file is skipped with a warning while `format:check` still exits 0, so crossing would quietly stop checking the snapshot rather than fail on it. The ceiling is raised for that one file through an override; a global raise would also disarm the guard for hand-written files, which is what it exists for. The refresh also flips 15 ZenMux models from active to deprecated. Two carry a model-level protocol override and are excluded from the fallback source, so 13 reach it. Discovery keeps only the fallback set and the catalog marks what survives available and default-capable, so all 13 would have been offered as usable choices for a provider that had none before. ZenMux now filters deprecated ids at the call site, as six other providers already do. That fix is deliberately narrow. Six providers still ship deprecated ids in their fallback lists — 28 models across `togetherai`, `mistral`, `xiaomi`, `deepinfra`, `nvidia`, and `openai`, the last writing its list by hand rather than through `toolCallingModelIds`. Reverting this refresh changes none of them, so converging them is tracked in #3355, and a new contract test records that set as an explicit boundary which fails both on regression into it and on a listed provider being cleaned up without being removed. Not addressed here: persisted user catalogs keep ids this refresh retires, because nothing reconciles `enabledModelIds` against the current registry and `opencode-free` has no discovery contract that could later repair it. That predates this branch — #2050 removed `hy3-free` the same way and touched no persistence — and reverting this refresh does not fix it, since the ids stay retired upstream. Tracked in #3354. The four surviving OpenCode Free pins are not evidence of anonymous entitlement and are not claimed as such. Individual ids have had operational attention — #1720 called `big-pickle` anonymously, #2166 replaced an unhealthy bootstrap model — but the four have never been verified as a set, and this refresh adds no verification and no reachable model; it only drops two the upstream retired. Reviewed by @hqhq1025 with Codex assistance under the maintainer-approved review workflow; two findings were raised, one fixed here and one filed as #3354. Generated-by: Claude Code
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes #1858.
models.dev declares per-model
reasoning_options, butscripts/sync-model-metadata.mjsonly read thereasoningboolean, so the generated metadata could never carry effort options. Thinking controls were hand-declared per provider path inSTATIC_MODEL_METADATA, and the wire was a second hand-maintained switch inbuildProviderOptions. Both silently missed most providers (42 of 44 with reasoning models had at least one gap), so reasoning models showed no thinking knob and, even when forced, sent nothing.The fix makes models.dev
reasoning_optionsthe single source of truth for both the knob and the wire:toMetadatanow normalizesreasoning_options(effort values + toggle) intothinkingOptionsin the generated metadata. The existing static-over-generated merge inlookupModelMetadatameans the UI andthinkingOptionsForModelneeded zero changes.budget_tokensis recognized but not carried (no wire consumer yet); unknown option types fail loudly instead of silent drift.buildProviderOptionsfalls back to a wire-family mapper keyed on the resolved adapter kind — the same objectgetAIModelswitches on, including per-model models.dev npm package overrides. Namespace derivation is shared withgetAIModelviaopenAiCompatibleProviderOptionsNameso the two cannot drift. The variant gate (level is defined only when metadata declares it) is what makes the generalization safe.thinking-wire-contract.test.ts): every declared thinking variant across the full generated model universe must wire a real, non-empty shape, and declaredofflevels must map to a real off wire — the silent "forgot" state now fails CI.thinkingOptions).opencode-freepin updated for the now-deprecatedhy3-free; vision characterization tests re-pinned to models still present.offBehavioradapter knowledge, Maka-verified deltas) stay.Commit 2 is a pure refactor: the eleven duplicated
reasoningEffortone-liner cases collapse into the family fallback (reachable behavior identical — verified), and the identityopenaiCompatibleNamespacehelper is deleted.Commit 3 addresses three independent deepseek-v4-flash reviews:
['none','default']— qwen3.6-27b's value set misapplied; Groq docs and the repo's own verified comment say qwen3-32b has no reasoning_effort. Pinned to no options until a live check proves otherwise.off.Commit 4 addresses a second review round (gpt-5.6-sol, k3-256k):
lookupModelMetadata's alias rules and enumerated fallback models only, so alias drift or static-only models (e.g. theclaude-subscriptionderived table) could silently shrink coverage. Core now exportsmodelMetadataIdsForProvider(generated snapshot + static overrides under the same alias rules) and the sweep uses it.model-factory-thinking.test.ts: it asserted the same invariant as the sweep with a weaker JSON-substring off-wire check and is fully superseded.deepseek,stepfun-ai-step-plan,xai) and the now-duplicatetencent-token-plan/hy3pin.anthropic-messages/openai-responseswires), unreachable by the sweep without account models.Verification
scripts/sync-model-metadata.test.mjs: transform-level tests for reasoning_options passthrough (effort/toggle/budget_tokens), malformed shapes, unknown-type tripwire.thinking-wire-contract.test.ts: issue case + full-universe wireability invariant.model-factory-thinking.test.ts: family-fallback deepEqual pins (opencode override paths, zenmux, github-copilot, cohere native vs compat wires), groq pin characterization.npm run format:check,npm run lint: clean.mainin this environment (pre-existing, unrelated).reasoning_effort(dose-response verified, invalid value 400s with the enum list), so the opencode-go wire is real, not speculative.Review focus
reasoning_optionsnow drives both UI and wire. Where verified adapter knowledge contradicts upstream (groq qwen3-32b), a static pin suppresses with a documented reason; the contract test keeps suppressions honest.reasoning_effortfor every upstream-declared effort model. If any gateway rejects the param, it becomes a static suppression entry (documented, test-enforced) rather than a silent gap.toggleis carried through but still has no production consumer (off wires are gated onoffBehavior/none); a future real off switch can consume it.