fix(drift): classify gpt-transcribe / gpt-live-transcribe as excludeFamilies (wrong modality) by copilotkit-devops-bot[bot] · Pull Request #343 · CopilotKit/aimock

copilotkit-devops-bot · 2026-07-29T07:00:51Z

Classifies the two OpenAI model families that shipped 2026-07-28 — gpt-transcribe and gpt-live-transcribe — and pins the registry surfaces that classification touches.

The decision

Both families are transcription-only. gpt-transcribe serves /v1/audio/transcriptions and realtime transcription sessions; gpt-live-transcribe is realtime-only. Neither answers on /v1/chat/completions.

excludeFamilies.openai (src/__tests__/drift/model-registry.ts) — the text-listing canary. Excluded, because they are the wrong modality. This mirrors the existing entries for gpt-4o-transcribe, whisper-1 and gpt-realtime*, and follows established prior art for keep-but-not-live families (gemini-pro, gpt-3.5).
knownVoiceModelFamilies (src/__tests__/drift/voice-models.ts) — the voice/realtime canary, a deliberately disjoint registry. Added, because that is the remediation the collector itself prescribes: "Unknown realtime model detected — add to knownVoiceModelFamilies".

These are two different registries with two different jobs. Excluding from the text canary is not the same as silencing detection — the voice canary is where these families are tracked, and membership there is the tool's own sanctioned resolution of the finding it raised.

The Decision: lines in the two drift-proposals/ notes are deliberately not set to include. The drift-sync auto-apply path writes exclusively into includeFamilies, so following the note's own boilerplate instruction would misclassify both families as chat-mocked. The decision is recorded as prose; wiring an exclude decision value is tracked separately.

Live evidence

pnpm test:drift against the real provider APIs on this branch: 136 passed, 0 failed.

Both canaries that are red on main are green on this branch:

✓ live /models contains no unclassified family
✓ canary: GA realtime models available

So this branch alone resolves the drift. drift-live-pr passes in CI. Note that the drift job itself is gated if: github.event_name != 'pull_request' and therefore skips on PRs — a live Drift Tests dry-run on the final sha is the pre-merge gate, confirming the two families no longer flag, the realtime canary is in the expected state, and nothing else started flagging.

What is in this PR, stated plainly

8 commits, 9 paths. It is not a small diff: roughly 200 lines are pin machinery (logic-pin.test.ts is +193/-1), with the classification itself being a handful of lines. Included deliberately rather than deferred, because two of these pins were mandatory findings from review:

excludeFamilies.openai DATA_FROZEN re-pin — required by the membership change.
Pins for knownVoiceModelFamilies, gaRealtimeModels and isVoiceModelId, plus the two functions that consume the seed sets. Before this, adding a family to the voice registry tripped no guard at all, and neutering detectVoiceModelDrift to hasGA = true — which permanently disables the "provider removed a GA realtime family" alarm — passed the entire 4,788-test suite with nothing watching it.
An anchored assertion so the live text-lane canary cannot be silenced by neutering its wrapper.

All pins are ^-anchored and use behavioural anchors alongside checksums: a checksum reports "something changed", a behavioural anchor reports "this no longer detects an unclassified family". Every one was proven by mutation — the guarded value was broken, the guard observed failing, the mutation reverted, green confirmed. A pin nobody has watched fail is not a guard.

Also: un-mangles the src/__tests__/ paths cited in the two notes (Prettier had eaten the underscores, so the committed audit trail cited files that do not exist), adds drift-proposals/ to .prettierignore so that cannot recur, and drops two gemini-1.5 notes whose decision was already applied in a3dc250.

This PR touches zero files under scripts/. That is load-bearing: it means the change cannot alter delta keys, so no base/head skew and no transitional gate failure is possible.

Split out of this PR

This branch previously carried 19 commits. The collector, delta-keying and test-tooling work has been moved out, because three review rounds each found real mandatory defects and almost all of them were in guards the review loop itself had added — a scope signal, not a quality signal.

test(drift): drift guard and collector hardening (split out of #343) #349 (draft) — drift guard/tooling hardening: collector remediation strings, delta-key provenance, the remediation-strings guard, and the transition-window documentation. Expected red on the two canaries inherited from main until this PR lands.
A third PR, still to open — a confirmed fail-silent defect in scripts/drift-delta.ts: the delta key omits scenario, so same-provider entries collapse and genuinely new drift is downgraded from block to advisory. Demonstrated on a real run: block: [] with a new-in-head critical sitting in advisory, gate passing. Pre-existing, independent of this change, and more serious than anything in either of the other two PRs.

Recovery record

Pre-strip head: 146cb9d2b81017097480b0616d36b576411aa11c (19 commits from the merge-base). The guard commits were pushed to drift/guard-hardening before this branch was rewritten, and the force-push was lease-pinned to that sha. Nothing was destroyed.

This PR is deliberately not closed: drift-sync's dedup is open-PR-only, so closing it would re-arm the daily needs-human PR this thread exists to resolve.

Round-4 additions — the classification is now actually tested

Review found that the two families this PR exists to classify had no behavioural coverage at all: remove them and only the membership checksum reddened. A checksum reports "something changed"; it does not report "these two are excluded and gpt-live is not." Four tests now assert what the classification means, and they live in the default suite — so they gate every PR rather than only running when someone triggers the live drift job:

dropping gpt-transcribe reds its own test; dropping gpt-live-transcribe reds its own test
a substring-style exclude match reds the boundary on gpt-live
a prefix-style match reds it in reverse on gpt-live-transcribe-mini

That boundary matters concretely: gpt-live and gpt-live-transcribe are different families, and a substring bug of exactly that shape was found elsewhere in this repo during review (a provider-label fallback resolved "Gemini Live Transcription session" to Transcription).

The live-canary anti-silence work has been REMOVED from this PR and moved to #349. An earlier revision of this branch carried an anchor intended to stop the live text canary being silenced. It was defeated at four successive surfaces — the detector function, the gate deciding whether it runs, the call site feeding it, and finally the fetcher the leg calls — each fix closing the named surface and leaving another open one frame further out. One of those defeats passed because the assertion matched a substring that vitest's own diff output supplies, so it held whether or not production code produced it. Text-span pinning cannot close that class by extension, so it is out of this PR rather than hardened a fifth time. The replacement — a fetch-stubbed harness that injects a family, drives the real chain, and asserts a report is emitted — belongs in #349, where silencing at any frame fails it.

FROZEN extraction is lazy. It ran at module scope, so one renamed surface threw at import and took out every other pin, every behavioural anchor and the whole frozen-data block with it — 0 of 37 tests running. The obvious repair a future reader reaches for is deleting the offending entry, which would leave everything else silently unguarded. Failures are now contained to their own case.

Live provider canaries no longer run in the offline suite. assertNoUnclassifiedFamilies moved to a plain text-drift.ts with no describe blocks, so importing it no longer executes live specs. pnpm test is 4767 passed / 0 skipped, and running the full offline suite with a deliberately invalid key produces no request to api.openai.com. This leak was pre-existing — one offline test already imported the live module before this branch existed — and is only partly this PR's to fix; what is fixed is that nothing in the offline suite reaches the network at this head.

Also corrected: the frozen-surface inventory was wrong in both directions, an anchor claim covering 6 of 12 entries was narrowed to what it actually covers, and a test titled "still propagates" now actually calls the function it claims to exercise.

Deferred, deliberately: other offline regressions still living in models.drift.ts run only under test:drift. Moving them is a ~440-line change and out of scope here — that kind of accretion is what required this PR to be split in the first place.

Round-5

Anti-silence anchor removed and moved to test(drift): drift guard and collector hardening (split out of #343) #349, per the reasoning above. logic-pin.test.ts goes 37 → 29 tests. Three pin hashes deleted, zero added — nothing was re-pinned on the way out. text-drift.ts's docstring now records that wrapper as knowingly unguarded pending test(drift): drift guard and collector hardening (split out of #343) #349 rather than silently dropping a protection claim.
The mirror-equivalence guard now catches all three divergences its own docstring names. Two of the three previously passed when broken — no fixture contained a non-model token, and every output happened to already be sorted, so dropping either protection left the suite green. Two fixtures fix it, each with its own demonstrated failure.
The four behavioural classification tests still bind, re-verified on this head: dropping either family reds its own test, a substring match reds on gpt-live, a prefix match reds on gpt-live-transcribe-mini.

Noted for a follow-up, not done here: a single import { expect } from "vitest" at module scope in text-drift.ts is the only thing preventing scripts/drift-sync.ts from importing the real module. Replacing it with a plain Error carrying the message the collector parses would retire the ~440-line hand-maintained mirror entirely. One line out, 440 deleted.

Also pre-existing, reported not fixed: the reference scan excludes only __tests__/drift, so isFamilyStillReferenced returns true for never-existing families off fixture literals alone. It errs toward over-blocking a removal, so it is fail-safe rather than dangerous.

pkg-pr-new · 2026-07-29T07:01:30Z

Open in StackBlitz

npm i https://pkg.pr.new/@copilotkit/aimock@343

commit: c89cf6a

…ilies (wrong modality) OpenAI shipped two transcription families on 2026-07-28. `gpt-transcribe` serves /v1/audio/transcriptions plus realtime transcription sessions; `gpt-live-transcribe` is realtime/WebSocket transcription only. Neither responds on /v1/chat/completions, so neither is text-generation drift. Two independent listing canaries were both red on them: - models.drift.ts (via includeFamilies ∪ excludeFamilies) reported `Unclassified model family "gpt-transcribe"` / `"gpt-live-transcribe"`. - ws-realtime.drift.ts (via knownVoiceModelFamilies, whose isVoiceModelId regex matches /transcribe/) reported UNKNOWN_REALTIME_MODELS=gpt-transcribe,gpt-live-transcribe. Add both families to excludeFamilies.openai's transcribe cluster (alongside gpt-4o-transcribe / whisper-1 / gpt-realtime-*) and to knownVoiceModelFamilies' transcription group. The two sets are deliberately disjoint surfaces, so both need the entry. Re-pin the excludeFamilies.openai DATA_FROZEN membership hash accordingly — the sorted-membership delta is exactly these two additions and nothing else. Record the decision on the two drift-proposals/ notes PR #343 carries. The decision is EXCLUDE, written in prose rather than via drift-sync's automated `Decision: include` marker: that path calls addFamilyLiteralInSource(..., "includeFamilies", ...) exclusively, so using it would have put these in includeFamilies and mismarked them as families aimock mocks on the chat surface. Does not touch src/ws-realtime.ts's runtime serve-gate — that is PR #345.

…erge order The comment said `gpt-transcribe` "serves /v1/audio/transcriptions + realtime transcription sessions". That is true of OpenAI and false of aimock — driving the real server proves it: REJECTED gpt-transcribe -> Model gpt-transcribe does not support session type transcription REJECTED gpt-live-transcribe -> Model gpt-live-transcribe does not support session type transcription ACCEPTED gpt-4o-transcribe -> session.updated type=transcription ACCEPTED gpt-realtime-whisper-> session.updated type=transcription (src/ws-realtime.ts's transcriptionModels allowlist; the last two are controls showing the probe discriminates.) A reviewer read the comment as a statement about aimock and reached the wrong conclusion about this registry entry. Rewritten to be order-independent: it states what is true of OPENAI's API (both are audio→text surfaces, neither answers on /v1/chat/completions, so neither can be text-generation drift) and what membership in this set MEANS (a classification for the `/models` listing check). It deliberately makes no claim about what aimock serves in EITHER direction — "aimock does not serve these" would go stale the day something adds them, so the comment must not be falsifiable by a merge at all. Also: - The bare "2026-07-28" disagreed with the notes' "Detected: 2026-07-29". Both are correct about different events, so each date now names its event — OpenAI's ship date and the drift run's first observation — instead of one winning. - Records the EXCLUDE decision with its two note files and PR #343, which the entry previously cited nowhere. - Fixes a directional reference: "the exclude-by-rule patterns above" pointed at PREVIEW_FAMILY / GEMMA_FAMILY, which are 200 lines BELOW it.

…ilies (wrong modality) OpenAI shipped two transcription families on 2026-07-28. `gpt-transcribe` serves /v1/audio/transcriptions plus realtime transcription sessions; `gpt-live-transcribe` is realtime/WebSocket transcription only. Neither responds on /v1/chat/completions, so neither is text-generation drift. Two independent listing canaries were both red on them: - models.drift.ts (via includeFamilies ∪ excludeFamilies) reported `Unclassified model family "gpt-transcribe"` / `"gpt-live-transcribe"`. - ws-realtime.drift.ts (via knownVoiceModelFamilies, whose isVoiceModelId regex matches /transcribe/) reported UNKNOWN_REALTIME_MODELS=gpt-transcribe,gpt-live-transcribe. Add both families to excludeFamilies.openai's transcribe cluster (alongside gpt-4o-transcribe / whisper-1 / gpt-realtime-*) and to knownVoiceModelFamilies' transcription group. The two sets are deliberately disjoint surfaces, so both need the entry. Re-pin the excludeFamilies.openai DATA_FROZEN membership hash accordingly — the sorted-membership delta is exactly these two additions and nothing else. Record the decision on the two drift-proposals/ notes PR #343 carries. The decision is EXCLUDE, written in prose rather than via drift-sync's automated `Decision: include` marker: that path calls addFamilyLiteralInSource(..., "includeFamilies", ...) exclusively, so using it would have put these in includeFamilies and mismarked them as families aimock mocks on the chat surface. Does not touch src/ws-realtime.ts's runtime serve-gate — that is PR #345.

drift-proposals/gemini-gemini-1.5-{pro,flash}-deprecated-referenced.md still read "Status: NEEDS HUMAN REVIEW", but their decision was made and applied back in a3dc250: both families moved from includeFamilies.gemini into excludeFamilies.gemini's retired-but-still-mocked bucket, where they remain. Verified against current main before removing. They carried no information beyond that commit message — no decision line, no rationale — so leaving them behind just makes a settled item look open every time someone scans drift-proposals/. drift-sync will not re-create them: the deprecation half only walks includeFamilies (which no longer names them) and the addition half only walks unclassified families (they are classified). Repoint the registry comment that cited the note filenames at a3dc250 so the audit trail does not dangle.

Prettier's markdown formatter reads `__tests__` in prose as bold emphasis and rewrites it to `**tests**`, so both gpt-transcribe notes cited `src/**tests**/drift/...` — files that do not exist. The notes are the audit trail for a hand-applied EXCLUDE decision, so the trail pointed at nothing. Backtick the two paths (verified to survive Prettier) and add `drift-proposals/` to `.prettierignore` so the whole class cannot recur: these notes are machine-generated records, not prose we style. Also checked whether Prettier disturbs drift-sync's `Decision:` marker (parsed line-anchored, scripts/drift-sync.ts:481). It does not bold or indent it — it only inserts blank lines around it — so no note was silently un-decided.

logic-pin.test.ts froze the text-classification surfaces and the includeFamilies/excludeFamilies membership, but the realtime canary's own seed data was pinned NOWHERE. Proven by mutation: adding "totally-bogus-family" to knownVoiceModelFamilies passed logic-pin.test.ts (20/20) AND the entire non-drift suite (4784 tests). A one-line edit could stop a live unknown voice family from ever being reported and nothing in the repo would notice — the exact silencing this file exists to prevent. Adds three pins in the existing style: - knownVoiceModelFamilies — sorted-membership hash, like the DATA_FROZEN include/exclude pins (the set legitimately grows, so membership not source) - gaRealtimeModels — same, and it gates the "GA family still exists" assertion, which a bogus entry would make satisfiable by a model that isn't there - isVoiceModelId — source checksum plus a behavioral anchor, like normalizeModelFamily: it decides which live ids the canary even looks at, so narrowing it makes the canary go quiet with nothing failing The new pins are ANCHORED, deliberately unlike the existing ones. `extract` uses String.match, which returns the FIRST hit, so an unanchored pattern can be satisfied by a decoy above the real surface — a commented-out copy of isVoiceModelId prepended to voice-models.ts makes the existing style pin sha 4454f35a… (the decoy) instead of f3d7f732… (the real function), leaving the real rule free to move. The new `extractSole` takes a `^`-anchored, m-flagged pattern and hard-errors on a second match rather than freezing whichever came first. The pre-existing pins keep their current extraction; retro-fitting them is a separate change. Each new pin was mutation-tested: bogus family in knownVoiceModelFamilies → that pin fails; bogus entry in gaRealtimeModels → that pin fails; isVoiceModelId widened with `|gpt` → both its checksum and its behavioral anchor fail. Green again with each mutation removed; voice-models.ts is byte-unchanged.

…erge order The comment said `gpt-transcribe` "serves /v1/audio/transcriptions + realtime transcription sessions". That is true of OpenAI and false of aimock — driving the real server proves it: REJECTED gpt-transcribe -> Model gpt-transcribe does not support session type transcription REJECTED gpt-live-transcribe -> Model gpt-live-transcribe does not support session type transcription ACCEPTED gpt-4o-transcribe -> session.updated type=transcription ACCEPTED gpt-realtime-whisper-> session.updated type=transcription (src/ws-realtime.ts's transcriptionModels allowlist; the last two are controls showing the probe discriminates.) A reviewer read the comment as a statement about aimock and reached the wrong conclusion about this registry entry. Rewritten to be order-independent: it states what is true of OPENAI's API (both are audio→text surfaces, neither answers on /v1/chat/completions, so neither can be text-generation drift) and what membership in this set MEANS (a classification for the `/models` listing check). It deliberately makes no claim about what aimock serves in EITHER direction — "aimock does not serve these" would go stale the day something adds them, so the comment must not be falsifiable by a merge at all. Also: - The bare "2026-07-28" disagreed with the notes' "Detected: 2026-07-29". Both are correct about different events, so each date now names its event — OpenAI's ship date and the drift run's first observation — instead of one winning. - Records the EXCLUDE decision with its two note files and PR #343, which the entry previously cited nowhere. - Fixes a directional reference: "the exclude-by-rule patterns above" pointed at PREVIEW_FAMILY / GEMMA_FAMILY, which are 200 lines BELOW it.

3287eed pinned the voice registry DATA. The functions that read it stayed unpinned, and the data pin cannot cover them: knownVoiceModelFamilies is built by mapping already-normalized seeds through normalizeVoiceModelFamily, so neutering the normalizer leaves the membership hash untouched. Three one-line silencing edits, each run against the full pin suite BEFORE this change: normalizeVoiceModelFamily -> `return id` → 24/24 pass detectVoiceModelDrift -> `unknown = []` → 24/24 pass detectVoiceModelDrift -> `hasGA = true` → 24/24 pass The third is the one that matters. Hard-wiring hasGA true permanently suppresses NO_GA_REALTIME_MODELS — the alarm for OpenAI renaming or removing the entire GA realtime family — and it passed the ENTIRE suite: 167/167 files, 4788 tests, zero failures. Nothing in the repo was watching it. The first two are caught today by ws-realtime-canary.test.ts (4 and 2 failures respectively), so they were never as unguarded as a bare pin-suite run suggests. Both functions now get an anchored `extractSole` source pin AND a behavioural anchor, because a checksum alone explains nothing and churns on reformatting. The behavioural cases assert what the functions must MEAN: snapshots collapse onto their family while a single-digit tail does not, a genuinely new family is reported as unknown, and hasGA is false when no GA family is present. Each of the three mutations now fails both layers, and the behavioural assertion fails on its own — it is not riding on the checksum. Also ran the live canary against the real OpenAI API afterwards: 3/3 green.

…cing `unclassifiedFamilies` (the detector) is already behaviourally covered — neutering it to `return []` reddens three tests in the default suite. Its ASSERTION WRAPPER was not: `assertNoUnclassifiedFamilies` is the only thing that turns a detected unclassified family into a `formatDriftReport` block for the collector's exit-2 lane, it runs only under the live `test:drift` config, and replacing its computed `unclassified` with a literal `[]` left the whole 4820-test default suite green — the text-lane twin of the `hasGA = true` hole. Anchored behaviourally (must throw, naming the family) rather than by checksum, with a fully-classified negative control so an always-throwing wrapper cannot satisfy it. Verified RED against the one-line mutation, GREEN after reverting.

`unclassifiedFamilies` / `assertNoUnclassifiedFamilies` / the C4 deprecation detector lived in `models.drift.ts`, which is a SPEC: its top-level `describe`s include the three LIVE `/models` canaries. Importing a spec executes its `describe`s, so every importer silently adopted them. Two files import these functions, and both paid for it: - `drift-sync-mirror-equivalence.test.ts` (pre-existing — the import is already there at merge-base 9ce3107) reported 35 tests, 29 of them models.drift.ts's, 3 of those LIVE; - `logic-pin.test.ts` (added in 2d43206) went from 29 tests to 56, running 22 of them a second time per cycle. Net effect: the LIVE canaries ran inside the OFFLINE `pnpm test` suite. With a key exported, `OPENAI_API_KEY=sk-fake npx vitest run src/__tests__/drift-sync-mirror-equivalence.test.ts` made a real round-trip to api.openai.com and failed on the 401 body. CI never hit it (no workflow runs `pnpm test` with keys), but any developer with a key exported did. Move the primitives to `text-drift.ts`, which registers NO suites, and point all importers there. `models.drift.ts` is now purely the spec. `text-drift.ts` still imports vitest's `expect` for the assertion wrapper, so `scripts/drift-sync.ts` still cannot import it from a plain `npx tsx` process — the reason its hand-maintained mirrors exist is unchanged, and the mirror-equivalence guard still holds the two copies to result-equivalence. Offline suite: 4767 passed, 0 skipped, no network. `logic-pin.test.ts` is back to 31 tests (its own 29 plus the two anchors 2d43206 added) with zero live legs, and the `unclassified = []` neuter still reddens exactly the anchor and nothing else. The offline regressions in `models.drift.ts` keep running under `pnpm test:drift`, which the drift workflow triggers on any `src/__tests__/drift/**` change.

… just that it moved The two families this PR exists to classify had ZERO behavioural coverage. Removing either from `excludeFamilies.openai` reddened only the `excludeFamilies.openai` membership CHECKSUM — which says "the data moved" and nothing about the classification. Neither family appeared in any `/models`-shaped payload in the suite, and most `excludeFamilies.openai` entries appeared in none either, so nothing proved a DATED snapshot of them (the form the live listing actually carries) collapses back onto the excluded key. Adds, in the default suite: - `gpt-transcribe` and `gpt-live-transcribe` are each EXCLUDED when they appear in a `/models`-shaped payload, bare and as a dated snapshot; - the `gpt-live` vs `gpt-live-transcribe` boundary, in BOTH directions. These are DIFFERENT families: the first is the canonical unclassified family the canary must keep reporting, the second an excluded transcription surface whose key has the first as a strict prefix. A substring/prefix-shaped classification bug would swallow the one family the canary exists for, and that exact shape has already been found in this codebase (a provider-label fallback where "Gemini Live Transcription session" resolved to `Transcription`); - every enumerated `excludeFamilies.openai` entry, bare and dated. Red-green, each behavioural test run targeted so the membership checksum is not what is being pointed at: - drop `gpt-transcribe` → "gpt-transcribe is EXCLUDED in a /models-shaped payload" fails (`expected false to be true`); - drop `gpt-live-transcribe` → its own test fails the same way; - make the exclude lookup a substring match (`f.includes(family)`) → the boundary test fails on `isClassifiedFamily("gpt-live")` returning true; - make it a prefix match (`family.startsWith(f)`) → the same test fails in the reverse direction, on `gpt-live-transcribe-mini` no longer being reported. Every mutation reverted; nothing re-pinned.

…n failures Three surfaces are now pinned for the live text canary, not one: 1. the CALLEE — `assertNoUnclassifiedFamilies`, already anchored behaviourally (it must throw, naming the family); 2. the GATE — `describe.skipIf(!process.env.<PROVIDER>_API_KEY)`. Rewriting it to `describe.skip` silences the canary permanently while the behavioural anchor stays green, both suites stay green, eslint and prettier stay clean, and the reported skip count does not even move (the leg was already counted as skipped whenever the key was absent); 3. the CALL SITE — `assertNoUnclassifiedFamilies(models, …)`. Passing `[]` (caught only by eslint's unused-var rule) or `models.slice(0, 0)` (eslint-clean) leaves the assertion running on nothing. Each of the three live legs is source-pinned WHOLE via `extractSole` with a `^`-anchored pattern, plus a readable structural anchor per leg so a red pin comes with a diagnosis instead of just "the bytes moved". Verified: with `describe.skip` the openai pin and its structural anchor both fail; with `models.slice(0, 0)` both fail again, the second on the "must pass the fetched models collection" message. Both mutations reverted. Separately, `FROZEN`'s `source` is now a THUNK. Extraction ran at MODULE scope, so one renamed or reshaped surface threw during COLLECTION and took the entire file with it — every other pin, every behavioural anchor and the whole `DATA_FROZEN` describe reported as "no tests". Renaming `DATED_SNAPSHOT_SUFFIX` with module-scope evaluation gives `Failed Suites 1` / `Tests no tests`; with the thunk, 2 cases fail (the missing surface, and `normalizeModelFamily` whose body genuinely changed) and the other 35 run and pass. Fail-loud is right for a moved surface; a total blast radius is not, because the obvious repair a future human reaches for is deleting the offending entry, which silently unguards the rest. Also corrects two false records in this guard: the header inventory omitted the `includeFamilies`/`excludeFamilies` membership pins and the `DATA_FROZEN` describe title omitted the voice seed sets it also freezes; and the `\n}` anchoring note claimed to describe every entry when it only applies to the function-body patterns, not the single-line consts or the set literal.

…349) This is NOT a coverage reduction to accept on its own terms — it is a mechanism that does not work being taken out of the way of one that does. Removed from logic-pin.test.ts: the behavioural anchor on `assertNoUnclassifiedFamilies`, its partnering known-negative control, and the per-leg source pins on each live canary's skip GATE and CALL SITE (`freezes liveLeg.{openai,anthropic,gemini}` plus their structural anchors, including the "must pass the fetched `models` collection" assertion). WHY, and why not a fifth pin instead. The live text canary is a CHAIN — fetcher -> gate -> call site -> formatter -> collector. Pinning the TEXT of one link only relocates the silencing edit one frame outward, and that has now happened four times against this same anchor: 1. the skip gate (`describe.skipIf(...)` -> `describe.skip`) 2. the call site (`assertNoUnclassifiedFamilies(models, ...)` -> `models.slice(0, 0)`) 3. the report FORMAT — replacing the `formatDriftReport(...)` payload with an opaque string. Verified locally: logic-pin stayed 37/37 green, because the surviving `.toThrow(/gpt-live/)` was satisfied by vitest's OWN diff text (`opaque drift message: expected [ 'gpt-live' ] to deeply equal []`) rather than by anything production emitted. The consequence is not cosmetic: the collector routes on `/API DRIFT DETECTED:/`, so real drift with an unformatted payload lands in exit-5 quarantine instead of the exit-2 auto-fix lane — seen, misrouted, never fixed. 4. the FETCHER — `providers.ts`'s `listOpenAIModels` `return json.data.map(m => m.id)` -> `.slice(0, 0)`. Full suite green, eslint and prettier clean, canary silenced end to end. A fifth defeat is already known: an example-shaped callee anchor is beaten by `.filter(f => f === "gpt-live")`, which keeps the one asserted family and drops every other. Each fix closed the named surface and left the next one open, so the DESIGN is what is wrong, not the coverage. The replacement, built in #349, is a fetch-stubbed end-to-end harness: stub the fetch layer to return a listing containing an unclassified family and require the chain to emit a collector-routable `API DRIFT DETECTED:` report. That observes the OUTPUT of the whole chain rather than the text of one link, so silencing at the fetcher, gate, call site or formatter all fail it. The probe built to demonstrate defeat 4 is the seed of that harness. Kept intact — this PR's actual subject and its supporting structure: the four behavioural classification tests in text-drift.test.ts, the `text-drift.ts` extraction itself (those tests depend on it, and it is what keeps the live canaries out of the offline suite), the lazy `FROZEN` thunk, the frozen-surface inventory correction, the narrowed anchor claim, and the "still propagates" fix. All remaining pins verify unchanged — the three removed live-leg hashes are the only `pin:` lines touched, and nothing was re-pinned. `assertNoUnclassifiedFamilies` stays exported (the three live legs of models.drift.ts call it) with its doc corrected: it now records the wrapper as knowingly unguarded pending #349, instead of claiming an anchor that no longer exists.

…ee named divergences The guard's docstring claims it protects against "a sort order tweak, a dropped `NON_MODEL_TOKENS` check, a changed fail-closed floor". Only the third was true — the other two were undetectable, because equivalence assertions are only as strong as the fixtures fed to them and no fixture DISTINGUISHED the two behaviours: - no fixture contained a `NON_MODEL_TOKENS` member (the set has exactly one, `gemini-interactions`), so the mirror's token-suppression line was dead weight as far as this test could tell; - every fixture's output happened to already be in insertion order, so the mirror's `.sort()` was equally invisible. Verified before the change: deleting the mirror's `NON_MODEL_TOKENS.has(...)` line left this file 6/6 green, and deleting its `.sort()` left it 6/6 green. A guard that documents three protections and delivers one is a false record — it invites a reader to assume the other two are covered. Two fixtures added, each chosen to distinguish exactly one of them: - `["gpt-4o", "gemini-interactions"]` — the token is unclassified for openai on its own merits, so the check is the only reason it is suppressed. With the mirror's check dropped this now fails: AssertionError: expected [ 'gemini-interactions' ] to deeply equal [] - `["gpt-omega", "gpt-alpha"]` — two unclassified families supplied in reverse-alphabetical order, so insertion order and sorted order differ and `toEqual`'s order-sensitivity bites. With the mirror's `.sort()` dropped this now fails: AssertionError: expected [ 'gpt-omega', 'gpt-alpha' ] to deeply equal [ 'gpt-alpha', 'gpt-omega' ] Both mutations reverted; `scripts/drift-sync.ts` is byte-identical to its committed state and no file under `scripts/` is modified by this commit. The docstring now also records that the fixtures are LOAD-BEARING rather than illustrative, with a warning not to prune apparent near-duplicates without first re-checking that the mutation each exists to catch still reddens. That is the failure mode that produced this gap in the first place.

copilotkit-devops-bot Bot assigned jpr5 Jul 29, 2026

jpr5 force-pushed the drift-needs-human/2026-07-29-30429968759 branch from 4c6a3c4 to ced241b Compare August 3, 2026 14:31

jpr5 changed the title ~~chore(drift-sync): needs-human model-family decision (2026-07-29)~~ fix(drift): classify gpt-transcribe / gpt-live-transcribe as excludeFamilies (wrong modality) Aug 3, 2026

jpr5 mentioned this pull request Aug 3, 2026

fix(drift): one Fix Drift run per day, and an alert that names the PR #348

Draft

aimock-drift-bot and others added 8 commits August 3, 2026 13:24

fix(drift-sync): mechanical model-family sync (needs-human note file(s))

7902717

jpr5 force-pushed the drift-needs-human/2026-07-29-30429968759 branch from 146cb9d to 2d43206 Compare August 3, 2026 20:37

jpr5 mentioned this pull request Aug 3, 2026

test(drift): drift guard and collector hardening (split out of #343) #349

Draft

jpr5 added 5 commits August 3, 2026 15:04

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fix(drift): classify gpt-transcribe / gpt-live-transcribe as excludeFamilies (wrong modality) - #343

fix(drift): classify gpt-transcribe / gpt-live-transcribe as excludeFamilies (wrong modality)#343
copilotkit-devops-bot[bot] wants to merge 13 commits into
mainfrom
drift-needs-human/2026-07-29-30429968759

copilotkit-devops-bot Bot commented Jul 29, 2026 •

edited by jpr5

Loading

Uh oh!

pkg-pr-new Bot commented Jul 29, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

copilotkit-devops-bot Bot commented Jul 29, 2026 • edited by jpr5 Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

The decision

Live evidence

What is in this PR, stated plainly

Split out of this PR

Recovery record

Round-4 additions — the classification is now actually tested

Round-5

Uh oh!

pkg-pr-new Bot commented Jul 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

copilotkit-devops-bot Bot commented Jul 29, 2026 •

edited by jpr5

Loading

pkg-pr-new Bot commented Jul 29, 2026 •

edited

Loading