fix(drift-sync): record provider deprecations instead of paging a human - #365
Merged
Conversation
A family a healthy live /models listing no longer contains is a fact the provider already published, not a decision. drift-sync now records it in a new deprecatedFamilies ledger in model-registry.ts and leaves includeFamilies (and therefore the mock) untouched, instead of writing a needs-human note that reddens the daily cron and emails the owner every morning.
…obe tests Adds coverage for the empty-array append, the ten-in-one-run shape, the second-run silence, and the pin staying green; adds the ledger's own registry invariants and the mirror's second bounded divergence. Drops the tests for the zero-reference removal probe along with the probe itself.
…r cannot void them Recording anthropic's ten 2026-08-07 retirements broke four fixtures that hard-coded claude-3-opus as a genuinely-retired stand-in: the mirror now correctly drops a recorded family, so those fixtures asserted on an empty candidate list. The stand-ins are derived from families that are neither forward-looking nor already recorded, and throw loudly if none remain.
commit: |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A model family that a provider's own live
/modelslisting no longer returns is a fact, not a decision. drift-sync was escalating every one of them to a human, reddening the daily cron and emailing the owner to rubber-stamp bookkeeping.Run
31218975992did exactly that: ten Anthropic families (claude-3-opus,claude-3-sonnet,claude-3-haiku,claude-3-5-sonnet,claude-3-5-haiku,claude-3-7-sonnet,claude-opus-4,claude-sonnet-4,claude-haiku-4,claude-opus-4-1), one PR, one red run, one email — with nothing for a human to actually decide.Why it escalated
Not judgement. A structural dead end.
includeFamiliesmembership is checksum-pinned, drift-sync's own gate re-runs that pin, and the allowlist forbids re-pinning it — so removing a retired family always reds its own gate and delivers nothing. With no place to write the fact down, the only reachable outcome was to ask a human.What changes
A
deprecatedFamiliesledger inmodel-registry.tsgives the fact somewhere to live. Recording a deprecation is now mechanical, and both classes collapse to the same action:A genuinely new or unclassified family still pages a human, unchanged. "Should aimock mock this?" is a real product question.
includeFamiliesis byte-identical — verified by AST extraction (SHA-256281ed637a2f5…, 2300 bytes both sides), and the pin test reds when a family is added.Verification
Red and green replay the real production input. The replay reproduces changeset key
9c868d12bd208698— the exact key from run31218975992— and its notes diff empty against PR #364, so this is the input that paged the owner, not a synthetic stand-in.main: tenneeds-human-*,reason=needs-human, exit 1.[deprecation-recorded], zero needs-human,reason=ok-applied, exit 0. All three gates ran and passed. A second run reportsok-no-churnand stays silent.The nine still-referenced mocks still serve: built
dist/, ran the real server, POSTed/v1/messagesper family — 9/9 HTTP 200 with real mocked bodies.Negative control: a synthetic new family
claude-zephyr-9still yieldsreason=needs-human, exit 1, and aDecision: pendingnote.The ledger cannot reach the serving path: zero hits in a fresh
dist/(createServerscores 42, proving the search works), no non-testsrc/file references the registry, and the boundary test reds when the symbol is added tosrc/constants.ts.Ten mutations, all red — including a stray ledger entry, unclassify-on-record, a ledger that swallows every fixture, and detect-but-never-record.
Along the way this surfaced four fixtures hard-coding
claude-3-opusas "genuinely retired", which would have reddened CI on the first post-merge drift PR. They now derive their stand-ins and throw loudly rather than pass vacuously.Unproven
Listings are replayed rather than fetched live; Anthropic fidelity is established by the changeset-key and note equality above, while openai/gemini inputs are synthesized. The ledger is append-only by choice: a resurrected family would keep a stale marker, traded against record/un-record PR churn.