fix(drift): surface known-models canary drift as critical instead of crashing#291
Merged
Merged
Conversation
commit: |
jpr5
force-pushed
the
fix/ws-realtime-drift-and-collector-parsing
branch
from
July 7, 2026 07:55
942baca to
c883248
Compare
…crashing The Fix-Drift collector crashed (run 28848422043) when OpenAI shipped new realtime models: the canary's vitest AssertionError could not be turned into a structured drift entry, so the run exited non-zero with no report. Collector + canary changes: - Add gpt-realtime-2.1 / -2.1-mini to the known-models set. - Have the canary emit stable custom-message markers (UNKNOWN_REALTIME_MODELS= and NO_GA_REALTIME_MODELS=) so the full model list survives vitest's array truncation; parseKnownModelsCanary parses them into a typed CanaryParseResult and collectDriftEntries emits a critical OpenAI-Realtime entry (exit 2) for both the unknown-model and GA-family-gone cases. - Fail loud on unrecognized unparseable failures instead of swallowing them as benign infra (a false exit-0 was the original incident), and uniformly anchor every infra indicator through one wrapper so a labelled drift value can no longer trip the infra gate. - Gate the canary's printed-array fallback to the ws-realtime context so a non-canary toEqual([]) failure from any provider is no longer misattributed as OpenAI-Realtime model drift. - Run the canary in CI on OPENAI_API_KEY (the /v1/models list works with any key); the real WS session tests keep requiring OPENAI_REALTIME_KEY. - Guard the module entry point so importing it for tests does not run main().
…a anchoring Replace the divergent local reimplementations of the collector functions with imports of the real exported functions, so the tests actually guard shipped behavior (the canary->critical->exit-2 path had no real coverage before). - Exercise parseKnownModelsCanary, classifyUnparseableAsInfra, parseDriftBlock, and collectDriftEntries directly. - Add a property-based test that iterates the real INFRA_INDICATOR list and asserts, per indicator, that a labelled `Real: <sample>` value is not classified as infra while a bare failure reason still is. - Cover the NO_GA / UNKNOWN marker shapes (including the real Unicode-ellipsis truncated form) and the non-canary fallback-gating case.
jpr5
force-pushed
the
fix/ws-realtime-drift-and-collector-parsing
branch
from
July 7, 2026 17:05
c883248 to
ec15911
Compare
jpr5
changed the title
jpr5
marked this pull request as ready for review
July 7, 2026 17:15
jpr5
added a commit
that referenced
this pull request
Jul 15, 2026
…crashing (#291) ## What this fixes The scheduled **Fix Drift** workflow crashed ([run 28848422043](https://github.com/CopilotKit/aimock/actions/runs/28848422043)) the moment OpenAI shipped new realtime models. The known-models canary failed as intended, but the collector couldn't turn the vitest `AssertionError` into a structured drift report — so it hard-exited instead of opening an auto-fix PR. Two root problems, both fixed: 1. **New models weren't recognized.** Added `gpt-realtime-2.1` / `gpt-realtime-2.1-mini`, and — more importantly — the canary now emits a stable custom-message marker (`UNKNOWN_REALTIME_MODELS=` / `NO_GA_REALTIME_MODELS=`) carrying the full model list, so the collector parses it deterministically instead of scraping vitest's truncated array output. Both failure modes (a new unknown model, or the GA family being renamed/removed) now produce a **critical** entry and exit 2 → auto-remediation. 2. **The collector mis-handled unparseable failures.** It now **fails loud** on anything it can't positively classify (a false "no drift, exit 0" was the original incident class); every infra indicator is uniformly line-anchored through one wrapper so a drift value like `Real: API returned 503` can't masquerade as an infra error; and the canary's printed-array fallback is scoped to the realtime context so a non-canary `toEqual([])` failure from another provider can't be misattributed as realtime model drift. ## Testing Every fix was proven red→green through the **real exported collector functions** against **real** vitest `--reporter=json` output (not hand-authored strings). Full suite: **4327 passing**. The tests now import the real functions rather than reimplementing them, and a property-based test iterates the real infra-indicator list so no indicator can be individually mis-anchored again. ## Review Ran a full review-fix loop to convergence — zero mandatory findings on the final round, and a promotion audit confirmed no deferred item undermines the canary→exit-2 guarantee. ## Follow-ups (out of scope — separate work) - The HTTP drift run's vitest glob includes `agui-schema.drift.ts`, so AG-UI-only drift would exit-1 instead of exit-2 — needs `vitest.config.drift.ts`. - `fix-drift.yml`: auto-fix `continue-on-error` can leave a crashed fix silently unremediated; the Slack message's `\n` renders literally; benign exit-4 isn't special-cased. - Pre-existing collector-parse robustness (AG-UI phrasing regexes, CRLF handling in `parseDriftBlock`, provider-name matching). Draft until reviewed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this fixes
The scheduled Fix Drift workflow crashed (run 28848422043) the moment OpenAI shipped new realtime models. The known-models canary failed as intended, but the collector couldn't turn the vitest
AssertionErrorinto a structured drift report — so it hard-exited instead of opening an auto-fix PR.Two root problems, both fixed:
New models weren't recognized. Added
gpt-realtime-2.1/gpt-realtime-2.1-mini, and — more importantly — the canary now emits a stable custom-message marker (UNKNOWN_REALTIME_MODELS=/NO_GA_REALTIME_MODELS=) carrying the full model list, so the collector parses it deterministically instead of scraping vitest's truncated array output. Both failure modes (a new unknown model, or the GA family being renamed/removed) now produce a critical entry and exit 2 → auto-remediation.The collector mis-handled unparseable failures. It now fails loud on anything it can't positively classify (a false "no drift, exit 0" was the original incident class); every infra indicator is uniformly line-anchored through one wrapper so a drift value like
Real: API returned 503can't masquerade as an infra error; and the canary's printed-array fallback is scoped to the realtime context so a non-canarytoEqual([])failure from another provider can't be misattributed as realtime model drift.Testing
Every fix was proven red→green through the real exported collector functions against real vitest
--reporter=jsonoutput (not hand-authored strings). Full suite: 4327 passing. The tests now import the real functions rather than reimplementing them, and a property-based test iterates the real infra-indicator list so no indicator can be individually mis-anchored again.Review
Ran a full review-fix loop to convergence — zero mandatory findings on the final round, and a promotion audit confirmed no deferred item undermines the canary→exit-2 guarantee.
Follow-ups (out of scope — separate work)
agui-schema.drift.ts, so AG-UI-only drift would exit-1 instead of exit-2 — needsvitest.config.drift.ts.fix-drift.yml: auto-fixcontinue-on-errorcan leave a crashed fix silently unremediated; the Slack message's\nrenders literally; benign exit-4 isn't special-cased.parseDriftBlock, provider-name matching).Draft until reviewed.