Repository navigation
Releases
Releases · feronovak/llm-preflight
2.25.9
Added
gemini-4-neonnow has an official snapshot price, an active retirement row and tier 1 in the model importance list. Rates are flat across the 1M window, in USD per million tokens (input / output / cached input): 3.00 / 15.00 / 0.15, read from Google's pricing page on 2026-11-19. It takes the Gemini 4 request checks withXHIGHthinking, likegemini-4-hydrogen; the eight-case measured run is a follow-up. Breaking: configs that namegemini-4-neonchange results.--pricing-checkexits 0 instead of 1;--dry-runwithrequire_current_pricingormax_estimated_cost_usdexits 0 instead of 2; approval receipts recorded without a cost cap now reportmismatch; the retirement verdict isactiveinstead ofunknown, so--ciexits 3 once the row is stale (from 2026-12-20);--verify-approval-receiptexits 1 instead of 0 for approval receipts recorded with no cost cap. AcachedContentsetting, a thinking level outsideLOW/MEDIUM/HIGH/XHIGH(MINIMALis rejected) or a synchronous output cap over 65,536 now fails before any request (--dry-rundoes not check these). Sampling parameters are omitted, and catalog discovery no longer claims temperature support.
Changed
- Re-measured
glm-5.3,glm-5.3-flashx,glm-5.3-flashandkimi-k3.1-proon 2026-11-18 (eight cases each, same method as 2026-10-29) and updated the README table, comparison guide and results image.glm-5.3-flashximproved from 7/8 to 8/8; the other two GLM sizes are unchanged at 7/8; Kimi K3.1 Pro is still incomplete (3 responses and 5 HTTP 429 rate limits). The README now shows 225 of 232 checks passed and 26 of 29 models passing all eight.
2.25.8
Changed
- Added
gpt-6.2-astrato the measured-models comparison: an eight-case run on the nativeopenairoute, rendered into the results image and the README table. - All bundled price and retirement rows were re-checked against their official pages on 2026-11-18; prices, retirement dates and replacements are unchanged. The review dates moved, so the rows now become stale on 2026-12-19 (strict pricing, and
--ciexit 3 for retirement rows). Two groups keep their earlier date:gpt-5.6-solstays 2026-11-04 and becomes stale on 2026-12-05, because its promotional price runs to 2026-11-30 and the post-promotion price is re-read separately, and the rows reviewed on 2026-11-17 (claude-opus-4-5-20251101,claude-opus-4-6,claude-opus-4-7,claude-sonnet-4-6, the four DeepSeek rows and the four Qwen rows) stay stale on 2026-12-18. 55 OpenAI retirement rows that the deprecations page lists under Past deprecations (gpt-5.4-cyberand 54 older ones) are nowretiredinstead ofdeprecated; their shutdown dates had already passed, so no verdict changes. Breaking: every approval receipt, mock plans included, now verifies asmismatch(--dry-run --verify-approval-receiptexits 1 where it exited 0, andmigrate --rungates on the same receipt), because the retirement snapshot's oldest and newest review dates are part of every plan, and the pricing review date is part of any plan that names a re-dated price row; record them again with--dry-run --approval-receipt --replace-approval-receipt.
2.25.7
Added
gpt-6.2-astraandgpt-6.2-solnow have official snapshot prices, so a smoke or dry run reports a cost instead of "missing price" (USD per million tokens, input / output / cached input / 30-minute cache write, up to 272,000 input tokens): 8.00 / 40.00 / 0.40 / 10.00 for Astra and 2.00 / 10.00 / 0.10 / 2.50 for Sol; above that, 16.00 / 60.00 / 0.80 / 20.00 and 4.00 / 15.00 / 0.20 / 5.00. Both are rated tier 1 in the model importance list. Like the other GPT-6 models they omittemperature, accept only the standard service tier and 30-minute cache retention, and require a completed response status. Saved plans for these models change once because they are now priced. Breaking: for a config naming either model,--pricing-checknow exits 0 where it exited 1, and--dry-runexits 0 where it exited 2 when the config setsrequire_current_pricingormax_estimated_cost_usd. A config that setsservice_tierother thandefault,standardorauto, orprompt_cache_retention: "12h", for either model now makes--dry-runexit 2 where it exited 0 (a config that also sets a cost cap orrequire_current_pricingalready exited 2) and--doctorexit 1 where it exited 0, and a run stops before any request; remove that setting, then record approval receipts again with--dry-run --approval-receipt --replace-approval-receipt. An approval receipt recorded before this change for a config naming either model with no cost cap and norequire_current_pricingnow fails--dry-run --verify-approval-receiptwithmismatch(exit 1 where it exited 0) because the plan is now priced; record it again with--replace-approval-receipt.gate(CLI and MCP) now returnsroutes_changed(exit 1 where it exited 0) for a passing CI record attested before this change for a config naming either model with owner prices, because the route fingerprint now records thattemperatureis not sent; attest a new CI record. A Responses reply for these models without astatusnow fails the request.
Changed
- Added
gpt-6.2-solto the measured-models comparison: an eight-case run on the nativeopenairoute, rendered into the results image and the README table. - The four DeepSeek rows (
deepseek-flash,deepseek-v4-flash,deepseek-v4-flash-vision-exp,deepseek-v4-pro) were re-reviewed against the official pricing page on 2026-11-17; their prices are unchanged and they now become stale on 2026-12-18. The legacydeepseek-v4-flashnames bill at the Flash price. Breaking: approval receipts recorded for a DeepSeek plan before this change verify asmismatch(--dry-run --verify-approval-receiptexits 1), because the pricing review date is part of the plan; record them again with--dry-run --approval-receipt --replace-approval-receipt. - The pricing review date for
qwen3.8-max,qwen3.8-flashandqwen3.7-plusis now 2026-11-17, re-checked against the official Model Studio pricing page. Prices are unchanged, so their snapshot stays fresh through 2026-12-17.
2.25.6
Added
claude-opus-4-5-20251101,claude-opus-4-6,claude-opus-4-7andclaude-sonnet-4-6now have official snapshot prices, so a smoke or dry run reports a cost instead of "missing price" (USD per million tokens, input / output / cached input): 5.00 / 25.00 / 0.50 for the three Opus models and 3.00 / 15.00 / 0.30 for Sonnet 4.6. No cache-write rate is bundled, as for the other Anthropic rows. Saved plans for these models change once because they are now priced.claude-opus-4-5-20251101does not supportinference_geo, so a plan that sets it is refused before any request, as forclaude-haiku-4-5-20251001. Breaking: for a config that setsinference_geoforclaude-opus-4-5-20251101,--dry-runnow exits 2 where it exited 0,--doctorexits 1 where it exited 0, a run stops with exit 2 before any request (it used to send one the API rejects with HTTP 400), and the MCPdry_run_plantool returns an error. To keep such a config runnable, removeinference_geofor this model, then record receipts again with--dry-run --approval-receipt --replace-approval-receipt. The four priced models also change exit codes:--pricing-checknow exits 0 where it exited 1, and--dry-runexits 0 where it exited 2 when the config setsrequire_current_pricingormax_estimated_cost_usd.
Fixed
claude-opus-4-7no longer fails withHTTP 400: temperature is deprecated for this model: default requests now omittemperaturefor it, and so does atemperatureset in the run request, like the other Anthropic models that reject it.
Changed
- Added
qwen4-maxto the measured-models comparison: an eight-case run on the nativeqwenroute, rendered into the results image and the README table.
2.25.5
Added
qwen4-maxnow has an official snapshot price on the nativeqwenroute: 1.50 / 4.50 USD per million tokens (input / output), cached input 0.075 and cache write 1.875 (5-minute rate). Saved plans for this model change once because it is now priced.qwen4-maxon the nativeqwenroute now sendsmax_completion_tokensby default, so the output limit covers reasoning plus the answer, as for Qwen 3.8. Breaking: aqwen4-maxconfig withmax_tokens_parameter: max_tokensand thinking not explicitly disabled is now refused before any request, with a message naming the constraint:--dry-runexits 2 where it exited 0,--doctornow reports the model as failed and exits 1, where it exited 0, and a run exits 2 before any request where it sent the request and could exit 0 or 1. To keep such a config runnable, setmax_tokens_parametertomax_completion_tokens(the default) or explicitly disable thinking.
2.25.4
Fixed
mistral-large-4andmistral-large-4-0withtemperature: 0and notop_p: 1are now refused before any request, with a message naming the constraint, instead of ending as an HTTP 400 at run time. Setprovider_options.mistral.top_pto 1 or omittemperaturein the run request. Theinit --providerstarter and the catalog watch request now set thattop_p, so they pass for Mistral Large 4. Doctor and--dry-runapply the default request when a config has norequestblock, as a run does. Breaking: for such a config,--dry-runand a run now exit 2 before any request, where--dry-runexited 0 and the run exited 1 with the HTTP 400 case failed;--doctornow reports the model as failed and exits 1, where it exited 0. This also applies to a config with norequestblock and to the top-levellow-latencyandstructuredpresets, which settemperature0 and are refused too unlesstop_pis 1. The built-in tests that use thestructuredpreset (structured-output-check,strict-json-extraction,code-patch-summary,source-grounded-quiz, and theagent-smokesuite that contains them) are refused even withtop_p1, because they replaceprovider_options:--dry-runexits 2 where it exited 0, a run exits 2 where it exited 1, and--doctorwithprofilesexits 1 where it exited 0. To keep a configuration runnable, setprovider_options.mistral.top_pto 1 or omittemperaturein the run request; to test this model, use a custom prompt with its own request instead of those built-in tests. No request fingerprint changes.
2.25.3
Added
- Six models now have official snapshot prices, so their cost is no longer unknown (USD per million tokens, input / output / cached input):
gpt-6-cyber20.00 / 100.00 / 2.00 (cache write 25.00; above 272,000 input tokens 40.00 / 150.00 / 4.00, with no published long-context cache-write rate, so a budget for such a prompt that may write the cache stays unknown),gpt-5.6-cyber12.50 / 75.00 / 1.25 (cache write 15.625),gemini-4-hydrogen1.50 / 7.50 / 0.075, andgrok-4.20-0309-reasoning,grok-4.20-0309-non-reasoningandgrok-4.20-multi-agent-03091.25 / 2.50 / 0.20 (2.50 / 5.00 / 0.40 from 200,000 input tokens). Saved plans for these models change once because they are now priced.gpt-5.6-cybersupports only the Responses API, so a run needsadapter: openai_responses. gemini-4-hydrogenis now in the measured-models comparison: all eight support-routing cases passed on the native Gemini route for $0.007221, and it appears in the README image and table, the comparison guide and the snapshot pages.
Changed
gemini-4-hydrogengets the Gemini 4 request checks, with its own thinking levels:LOW,MEDIUM,HIGHandXHIGHare accepted andMINIMALis rejected. As forgemini-4-argonandgemini-4-carbon, an explicitcachedContentand amax_output_tokensabove 65,536 fail before any request, ignored sampling fields are omitted, and discovery no longer reports temperature support. A thinking level other thanLOW,MEDIUMorHIGHongemini-4-argonorgemini-4-carbonstill fails.
Fixed
- The bundled replacements for four OpenAI models retiring on 2026-12-11 now match the official deprecations page:
gpt-5-2025-08-07ando3-2025-04-16namegpt-6.1-sol(wasgpt-5.6-sol),gpt-5-mini-2025-08-07namesgpt-6.1-terra(wasgpt-5.6-terra), andgpt-5-nano-2025-08-07namesgpt-6-luna(wasgpt-5.6-luna). The verdict,migrateandnext_commandshow the new replacement. The four rows were reviewed on 2026-11-13 and become stale on 2026-12-14. Breaking: approval receipts recorded before this change verify asmismatch(exit 1), becausenewest_as_ofin every dry-run plan is now 2026-11-13. Record them again with--dry-run --approval-receipt --replace-approval-receipt.
2.25.2
Fixed
--dry-run,--doctor, the interactive preview and the MCPdry_run_plantool now warn (owner_price_ignored) when an owner price field is replaced by the bundled official snapshot, for example a cached-only rate ongpt-4.1. Resolved prices and cost figures are unchanged. The interactive and--changed-sincedry-run paths carry the warning too, and a non-numeric owner value is named without being echoed. Breaking:--dry-run --verify-approval-receiptexits 1 (mismatch) for a receipt recorded for an affected configuration, where it exited 0 before, becausepricing_warningsgains an entry;migrate --rungates on the same receipt. Record such receipts again with--dry-run --approval-receipt --replace-approval-receipt.
Changed
- The agent guidance (the MCP
initializeinstructions andsafe-workflowresource, the Codex plugin skill, and the MCP and coding-agent guides) now tells an agent to reportpricing_warnings, includingowner_price_ignored, from a dry run. The configuration and pricing pages state that an ownercache_write_input_cost_per_millionapplies where the bundled row has no cache-write rate, anddocs/reference/results.mddescribespricing_warningsas covering ignored owner prices too.
2.25.1
Changes below are marked Breaking: where they can change an exit code.
Added
- The bundled retirement rows now include OpenAI
gpt-5.5-cyber-preview: deprecated, removal 2027-02-01, no replacement named on the official page.--audit-source --ciand--dry-run --cinow exit3for it (it wasunknown, exit0), and1from 2027-02-01. The row was reviewed on 2026-11-12 and becomes stale on 2026-12-13. Breaking: approval receipts recorded before this change verify asmismatch(exit 1), mock plans included, because the new row changesretirements.snapshot.rows(119 to 120) andnewest_as_ofin every dry-run plan. Record them again with--dry-run --approval-receipt --replace-approval-receipt(without--replace-approval-receiptthe command exits 2 when the receipt file exists).
Fixed
- Breaking: a run on
adapter: openai_responsesnow fails with the tool-call error when the response contains a tool call (such as afunction_callitem), instead of passing on any text beside it. Such a run exited 0 before and now exits 1. - Breaking:
--replaynow checks the saved result'ssource_configshape like every other config path: a wrong-typed field such aspresets: 7exits 2 and names the field instead of raising a traceback and exiting 1. - Breaking: estimates for cache hits on
gpt-4.1,gpt-4.1-mini,gpt-5.4-mini,gpt-5.4-nano,claude-fable-5,claude-opus-5andclaude-sonnet-5now use the published cache-read rates (USD per million tokens: 0.50, 0.10, 0.075, 0.02, 1.00, 0.50 and 0.20). They used the full input rate before, so estimates were too high. Cache writes stay unpriced for these models. Saved dry-run plans change once for configs using these models because the plan's model row gainscached_input_cost_per_million, so approval receipts recorded for them verify asmismatch(--dry-run --verify-approval-receiptexits 1, where it exited 0); record them again. An owner-setcached_input_cost_per_millionis kept only together with the owner's own input and output prices; otherwise the bundled rate now applies to these seven.
2.25.0
Changes below are marked Breaking: where they can change an exit code.
Added
- The retirement report now carries
snapshot.valid_until, the last day the rows of the selected models are not stale, in the doctor and dry-run JSON (the scan JSON carries the bundled-wide date). Text output of the doctor, dry-run and scan adds a notice during the last 7 days; it never changes the decision or exit code. Saved dry-run plans change once because the plan now includes the field, so approval receipts for the old plan verify asmismatch. - The Action guide explains how to keep the pin current with Dependabot.
- The MCP
safe-workflowresource, theinitializeinstructions, the plugin skill and the agent guides now tell an agent to reportretirement_snapshot.valid_untilfromaudit_sourceandretirements.snapshot.valid_untilfromdry_run_plan, and the CLI agent guides namealias_unmappedand the dry-run limit breach. The text is additive; no schema, tool, key or exit code changes.
Changed
- The README's measured-models image now shows all 25 summarized models in three result groups, ordered by provider inside each group and including Gemini 3.1 Flash-Lite, with a light variant, shorter columns and larger text, and the README gives the same rows as a collapsed table.
scripts/render_measured_models.pywrites both variants. - Breaking:
--audit-sourcenow reads whole model IDs in backtick strings in.js,.mjs,.cjs,.tsand.tsxfiles. A JavaScript or TypeScript project with a retired model ID in backticks now exits 1 under--audit-source --ciwhere it used to exit 0.--change-planuses the same scanner, so a changed JavaScript or TypeScript file with a backtick model ID now reports amodel_idtrigger instead ofcontract_surface. Backticks in comments count too, and template strings with${…}are still not matched.
Fixed
- Breaking:
--audit-source,--dry-run --ci,--doctor,migrateand the MCPaudit_sourceand
migrate_plantools now look model IDs up in lower case and resolve an Anthropic alias such as
claude-opus-4-1,claude-sonnet-4-0orclaude-3-7-sonnet-latestto the verdict of its dated snapshot row (the
verdict carriesmatched_id). Such an alias of a retired model used to beunknownand exit 0 under--ci;
it now exits 1. - Breaking: an OpenAI alias with dated snapshot rows but no row of its own (
gpt-5,gpt-5-mini,gpt-5-nano,
gpt-5-pro,o3,o3-pro, and the two search-preview IDs) is nowalias_unmapped(decisioninconclusive, with
alias_candidates) and exits 3 under--ci, where it exited 0.retirement_reportsummaries gain an
alias_unmappedcount andmigratereports thealias_unmappedoutcome. - The
--audit-source --ciexit 0 line no longer says "no referenced model is retired, retiring or stale" when IDs
are unknown. It says no referenced model has a retirement finding in the snapshot, and counts the model IDs that
have no snapshot row and are not verified. - Breaking:
--dry-run(alsocatalog watch-new --test --dry-run) now prints the plan and then fails with exit 2,
and the same message as the live run, when the config breaks a limit the live run refuses:max_requestsbelow
the possible requests,max_estimated_cost_usdexceeded or with unknown pricing, or incomplete pricing under
require_current_pricing. It used to exit 0 and the live run then refused. The plan, text and JSON, carries
budget_limitsand aLimit breach:line, and the MCPdry_run_plantool returnsok: falsewith the plan.
Approval receipts written by 2.24.2 verify asmismatchbecause the hashed plan gainsbudget_limits;
record them again with--dry-run --approval-receipt. Amax_requestsormax_estimated_cost_usdthat is not a
finite number is invalid input and is now refused with amust be a numbererror and exit 2, before any request,
in the dry run and the live run. A non-numeric value used to be a traceback, and"inf","Infinity"or"NaN"as
max_estimated_cost_usdran on 2.24.2 with no effective cost cap. - Breaking:
verifyandgateresolve an Anthropic alias route (for exampleanthropic:claude-opus-4-1) to its
dated snapshot row. A CI record whose checked route is a retired alias used to verify with exit 0 and pass the gate
asfresh_pass; it is nowsuperseded:verifyexits 3 andgateexits 1. - The
examples/visionandexamples/project-integrationconfigs bound retries withrequest.retry
max_attempts: 1, so their one-request cap holds on a live run. - Provider error text, saved results, reports and MCP output now mask the exact value of every environment variable
named byapi_key_env(eight characters or more) once a request or catalog discovery reads it, and a bare
Bearertoken of 16 or more characters. Keys without a known prefix (Mistral, MiniMax, Cohere, Z.ai,
openai_compatible) used to be echoed back unchanged when a provider error body repeated them. - The retirement guide now states the right review and stale dates, and lists what the source scan does not see. The Action guide says non-mock configs need their key variables set for the doctor step. The CLI reference documents
init --api-key-env,--write-env-exampleand--force,--replace-approval-receipt,catalog probe --ledger,pricing-refresh --provider, and--all-unapproved,--select-candidatesand--write-configforcatalog refreshandwatch-new. The decision reference notes that undated hand-typed prices only block underrequire_current_pricing.