fix(drift): key the delta by scenario — a new critical drift was landing in advisory

[jpr5]

The bug: fail-SILENT — a new critical drift was classified advisory and the gate PASSED

scripts/drift-delta.ts built the delta key as `${provider}::${id}` with const id = diff.id ?? diff.path. parseDriftBlock sets id: path, so the key is effectively provider + wire path — and scenario (the other half of a DriftEntry's identity) was absent from it.

The same wire path is asserted by many scenarios of one provider: openaiChatCompletionShape (non-streaming text) and the tool-call shape both carry usage.prompt_tokens (src/__tests__/drift/sdk-shapes.ts:35,81; scenarios "non-streaming text" / "non-streaming tool call", src/__tests__/drift/openai-chat.drift.ts:162,266).

So a drift already present on main in scenario A absorbed a genuinely new critical drift on the same field in scenario B: one key, base-present, routed to advisory, block empty, process.exit(0). No error, no annotation, required check green.

RED (real computeDelta, origin/main code)

base = [OpenAI Chat / non-streaming text / usage.prompt_tokens]
head = base + [OpenAI Chat / non-streaming tool call / usage.prompt_tokens, class critical]

block   : []
advisory: [{"provider":"OpenAI Chat","id":"usage.prompt_tokens","class":"critical"}]
fixed   : []
gate exit code would be: 0

GREEN (same inputs, after the fix)

block   : [{"provider":"OpenAI Chat","scenario":"non-streaming tool call","id":"usage.prompt_tokens","class":"critical"}]
advisory: [{"provider":"OpenAI Chat","scenario":"non-streaming text","id":"usage.prompt_tokens","class":"advisory"}]
fixed   : []
gate exit code would be: 1

Bound

It only bit when the base already carried that exact provider + path. A brand-new field, a new model id, or a provider with no base drift always keyed new and blocked correctly. That is why it never showed up as a false green in practice — it needed a pre-existing drift on main to hide behind.

Second defect: order-dependent last-wins collapse

A residual collision (two assertions in the same scenario reporting the same path) was resolved last-wins, so the class printed on a blocking key depended on report order. Real pre/post output, both orders, same provider+scenario+id:

=== PRE-FIX (origin/main): same provider+scenario+id, two classes ===
  [Critical, Advisory] block: ["advisory"]
  [Advisory, Critical] block: ["critical"]
=== POST-FIX ===
  [Critical, Advisory] block: ["critical"]
  [Advisory, Critical] block: ["critical"]

Routing never depended on class (the #292 invariant is untouched — routing is still key-presence only), but the annotation a human reads on a hard failure did. The most severe class now wins: critical > quarantine > advisory > none > (absent). CLASS_RANK is typed Record<DriftClass, number>, so adding a DriftClass member is a compile error until it is ranked.

Known-negative control — no false blocks

An unchanged base/head pair still blocks nothing; scenario scoping must not turn pre-existing drift into a block. Both scenarios survive as distinct advisory keys:

=== CONTROL: unchanged base/head ===
block   : []
advisory: [ non-streaming text / usage.prompt_tokens (advisory),
            non-streaming tool call / usage.prompt_tokens (critical) ]
fixed   : []

Covered by a named test (KNOWN-NEGATIVE control: an unchanged base/head pair still blocks nothing), plus a fixed-direction test so a scenario disappearing from head is reported rather than silently absorbed.

Cached base reports stay valid

The earlier key-shape incident in this area (gaModels → gaRealtimeModels) invalidated the cached base because the key lived in the collector's persisted diff.id. This change is different in kind: the delta gate imports drift-delta.ts from the PR checkout and indexes both reports with it (.github/workflows/test-drift.yml, "Delta gate" step), so a keyOf change applies symmetrically to base and head — no transitional block/fixed churn.

The requirement is that every key component exists in the persisted JSON. scenario is a required DriftEntry field, populated on every collector path since the pipeline's first commit (extractScenario falls back to the whole context string when there are no parens), so scenario-scoped keys are computable from any base report isBaseReportReusable accepts — and that guard already rejects anything not same-UTC-day. The transition note on isBaseReportReusable now records this distinction.

One follow-up for whoever owns .github/** (deliberately untouched here): the gate's annotation formats a key as ${k.provider} ${k.id}, so two blocked keys differing only in scenario print identically. DeltaKey.scenario is now available to include — additive, no behaviour change.

Gates

• pnpm build — clean
• pnpm test — 169 files, 4821 tests, 0 skipped, 0 failed (no timing-replay flake this run)
• pnpm lint — clean; prettier --check . — clean
• tsc --noEmit on both tsconfigs — clean; the delta-related test files typecheck standalone (the newly required scenario field surfaced one stale DeltaKey literal in the fix(drift): flag new voice/audio model families beyond the "realtime" substring #292 simulation stub, fixed)
• pnpm test:drift not run: this is a pure, side-effect-free reduction of two JSON reports with no live surface, and the offline suite drives the real computeDelta directly. Running it would spend live API calls for no additional signal.
• Version-neutral: no package.json, CHANGELOG.md, .claude-plugin/*, charts/*, packages/aimock-pytest/*, or .github/** file touched. git diff origin/main..HEAD | grep -E '^[-+].*pin:' is empty.

[pkg-pr-new]

Open in StackBlitz

npm i https://pkg.pr.new/@copilotkit/aimock@353

commit: 8361870