Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 19 additions & 13 deletions .github/workflows/fix-drift.yml
Original file line number Diff line number Diff line change
Expand Up @@ -178,11 +178,15 @@ jobs:
# drift-sync.ts fetches each provider's live `/models` listing directly (no
# drift-report.json input needed), diffs it against the frozen
# `model-registry.ts` classification, and:
# - a zero-reference deprecated family -> mechanical, comment-marked
# removal, gated behind drift-sync-check.ts (allowlist + checksum-pin
# re-assert + clean re-collect) BEFORE the edit is kept, then commits.
# - a still-referenced deprecation, or a genuinely new/unclassified
# family -> NEVER auto-edited; drops a family-keyed dedup note file
# - a deprecated family (of EITHER reference class) -> mechanical,
# comment-marked record in deprecatedFamilies, gated behind
# drift-sync-check.ts (allowlist + checksum-pin re-assert + clean
# re-collect) BEFORE the edit is kept, then commits. The provider's own
# listing already says the family is gone, so this is a fact to write
# down, not a decision to escalate — it NEVER reports needs-human, and
# the mock keeps serving (includeFamilies is untouched).
# - a genuinely new/unclassified family, or a registry structural
# mismatch -> NEVER auto-edited; drops a family-keyed dedup note file
# under drift-proposals/ and reports needs-human (the job goes RED
# below so a human sees it — and the note is already in the repo on
# subsequent runs, so a re-fire is not PR spam).
Expand Down Expand Up @@ -1555,10 +1559,9 @@ jobs:
echo "## Needs a human decision (drift-sync)"
echo ""
echo "The deterministic, zero-LLM drift-sync found a model-family change it must"
echo "NOT auto-apply (a genuinely new/unclassified family, a deprecation —"
echo "still-referenced or zero-reference — or a registry structural mismatch). It"
echo "wrote the note file(s) below and opened this PR so the decision is REACHABLE"
echo "in the repo."
echo "NOT auto-apply (a genuinely new/unclassified family, or a registry"
echo "structural mismatch). It wrote the note file(s) below and opened this PR so"
echo "the decision is REACHABLE in the repo."
echo ""
echo "> **Not auto-merged — a human decides.** For a *new-family* note: set the"
echo "> note's decision line to \`Decision: include\` (to classify it) or delete the"
Expand Down Expand Up @@ -1787,8 +1790,11 @@ jobs:
fi
echo "recorded the suppression ack on PR #${CLOSED_PR} — the delivered notice is not re-posted"

# Alert: the sync found a genuinely-new family or a still-referenced
# deprecation — the two irreducible human decisions. The note was persisted by
# Alert: the sync found a genuinely-new family, or could not locate the
# registry array it had to edit — the two irreducible human decisions. A
# DEPRECATION is deliberately not one of them: the provider's own listing
# already reports it, so drift-sync records it mechanically and this alert
# never fires for one. The note was persisted by
# the step above (a distinct PR was opened, or the note was already in the repo
# / already proposed in an open PR — no PR spam). This alert fires the job RED
# so a human sees it in CI status too.
Expand Down Expand Up @@ -1817,7 +1823,7 @@ jobs:
# A REAL newline — see the cancellation alert above for why a
# double-quoted "\n" renders literally in Slack instead.
NL=$'\n'
echo "::error::drift-sync found a genuinely-new model family or a deprecation it must not apply itself — needs a human decision (see the needs-human PR / drift-proposals/*.md)"
echo "::error::drift-sync found a genuinely-new model family, or a registry structural mismatch it must not repair itself — needs a human decision (see the needs-human PR / drift-proposals/*.md)"
if [ -z "${SLACK_WEBHOOK:-}" ]; then
echo "::error::SLACK_WEBHOOK not set — cannot send needs-human alert"
exit 1
Expand All @@ -1832,7 +1838,7 @@ jobs:
else
WHERE="No open PR carries this decision (the note may not have been persisted, or its PR was merged/closed). See the drift-sync-log artifact and \`drift-proposals/\`."
fi
MSG="🧭 *Drift sync — needs a human decision* — a new/unclassified model family or a still-referenced deprecated family was found. ${WHERE}${NL}Run: https://github.com/${REPO}/actions/runs/${RUN_ID}"
MSG="🧭 *Drift sync — needs a human decision* — a new/unclassified model family, or a registry structural mismatch, was found. ${WHERE}${NL}Run: https://github.com/${REPO}/actions/runs/${RUN_ID}"
PAYLOAD="$(jq -n --arg text "$MSG" '{text: $text}')"
# Bounded: `--max-time` so a hung POST cannot burn the job's (or a
# cancellation's) whole budget, and a small bounded `--retry` so a single
Expand Down
18 changes: 7 additions & 11 deletions DRIFT.md
Original file line number Diff line number Diff line change
Expand Up @@ -93,13 +93,9 @@ When a `critical` drift is detected:

## Model Deprecation

The `models.drift.ts` test scrapes model names referenced in aimock's test files, README, and fixtures, then checks each provider's model listing API to verify they still exist.
`models.drift.ts` normalizes each provider's live `GET /models` listing to family keys and subtracts the frozen classification in `model-registry.ts`. Two directions fall out of that subtraction: a live family we do not classify (**new family** — see the automated sync below), and a classified family the listing no longer contains (**deprecation**).

When a model is deprecated:

1. Update the model name in the affected test files and fixtures
2. Update `src/__tests__/drift/providers.ts` if the cheap test model changed
3. Run `pnpm test` and `pnpm test:drift`
**A deprecation needs nothing from you.** The daily sync records it in `deprecatedFamilies` and aimock keeps mocking the family, so clients pinned to a retired model id keep working. The only thing worth doing by hand is the cheap live test model: if `src/__tests__/drift/providers.ts` names a model that no longer exists, the live drift legs cannot run at all, so point them at a current one and re-run `pnpm test:drift`.

## Adding a New Provider

Expand Down Expand Up @@ -190,12 +186,12 @@ daily **scheduled cron** (independent of drift-test failure — a retired model
family does not, by itself, fail the drift tests):

1. **Sync** — `scripts/drift-sync.ts` fetches each provider's live `/models` listing directly and diffs it against the frozen classification in `src/__tests__/drift/model-registry.ts`:
- a classified family absent from live listings with **zero remaining aimock references** → the removal is mechanically determined but **not applied**: a family-keyed note under `drift-proposals/` proposes it and names the exact two-file edit. `includeFamilies`'s membership is checksum-pinned in `logic-pin.test.ts`, so removing a family means re-pinning it, and that re-pin is the reviewed decision the pin exists to force — a decision the sync's own changed-file allowlist forbids it from making. (A removal the sync applied itself would fail gate 2 on the pin, revert every note the same run wrote, and deliver nothing.)
- a still-referenced deprecated family, or a genuinely new/unclassified family → same route, for the stronger reason that the decision itself is a human's: a family-keyed dedup note file is written under `drift-proposals/` and the run is routed to a human (no PR spam on re-fire)
2. **Gate** — `scripts/drift-sync-check.ts` re-verifies any mechanical edit before (inside `drift-sync.ts`) and after (workflow defense-in-depth) it is kept: a changed-file allowlist (only `model-registry.ts` data literals + `drift-proposals/` notes), a checksum-pin re-assert over the frozen classification logic, and a clean re-collect
- a classified family a healthy live listing no longer contains → **a provider-confirmed deprecation is a fact, not a decision**, so it never routes to a human. drift-sync RECORDS it, mechanically, as a comment-marked entry in `deprecatedFamilies[provider]` (`model-registry.ts`), stamped with the date and with whether aimock's own source still references it. **The mock keeps serving**: `includeFamilies` is untouched, so every builder and fixture for that family still answers — users pin retired model ids in their own suites for years, and the upstream catalog shrinking is not a reason to break them. Recording it is also what makes it stop: the detector filters recorded families out of its candidate set, so the same retirement is not re-derived every morning for ever. Dropping a retired family from aimock altogether stays optional human cleanup (delete it from `includeFamilies` **and** `deprecatedFamilies`, then re-pin `DATA_FROZEN["includeFamilies.<provider>"]` in `logic-pin.test.ts`, all in one reviewed commit) — the re-pin is the reviewed decision the pin exists to force, which the sync's own changed-file allowlist forbids it from making. Nothing is broken while it is undone.
- a genuinely new/unclassified family, or a registry structural mismatch (the AST locator could not find the array it had to edit) → **not** auto-applied: the decision itself is a human's. A family-keyed dedup note file is written under `drift-proposals/` and the run is routed to a human (no PR spam on re-fire)
2. **Gate** — `scripts/drift-sync-check.ts` re-verifies any mechanical edit before (inside `drift-sync.ts`) and after (workflow defense-in-depth) it is kept: a changed-file allowlist (only `model-registry.ts` data literals + `drift-proposals/` notes), a checksum-pin re-assert over the frozen classification logic, and a clean re-collect. `deprecatedFamilies` is the one registry set deliberately **not** membership-pinned — a pin on the ledger the sync appends to would red on the sync's own append and revert it, every morning, forever. It gates no alert a human sees (`isClassifiedFamily` does not consult it), so there is nothing for a pin to defend; its invariants are asserted behaviourally in `model-registry.test.ts` instead.
3. **PR** — the workflow opens a pull request for a human to review + merge (never auto-merged), unless an open PR already proposes the same changeset or a human has already rejected it. There are two distinct PR classes:
- **`ok-applied`** — a successful mechanical registry edit, i.e. an **addition** a human already approved on a prior run (a deprecation never reaches this class; see above). Pushed onto the `fix/drift-*` branch `drift-sync.ts` committed onto; a human reviews CI + the diff and merges.
- **`needs-human`** — a routed decision. `drift-sync.ts` commits the `drift-proposals/` note file(s), and the workflow pushes a **distinct `drift-needs-human/*` branch** and opens a PR so the note lands in the repo (the job also goes RED + Slack-alerts so the decision is seen). The PR is **never auto-merged**. To approve a _new-family_ note, set its `Decision: include` line and **merge the PR**; the **next** drift-sync run reads the approved note from `main` and applies the mechanical registry edit (an `ok-applied` PR). That two-run hand-off is how the loop closes.
- **`ok-applied`** — a successful mechanical registry edit: a recorded **deprecation**, or an **addition** a human already approved on a prior run. Pushed onto the `fix/drift-*` branch `drift-sync.ts` committed onto; a human reviews CI + the diff and merges. No alert, no red run — it is data-only bookkeeping.
- **`needs-human`** — a routed decision, and now only a genuinely new/unclassified family or a registry structural mismatch. `drift-sync.ts` commits the `drift-proposals/` note file(s), and the workflow pushes a **distinct `drift-needs-human/*` branch** and opens a PR so the note lands in the repo (the job also goes RED + Slack-alerts so the decision is seen). The PR is **never auto-merged**. To approve a _new-family_ note, set its `Decision: include` line and **merge the PR**; the **next** drift-sync run reads the approved note from `main` and applies the mechanical registry edit (an `ok-applied` PR). That two-run hand-off is how the loop closes.

**Closing a drift-sync PR REJECTS that changeset, permanently.** A CLOSED-but-never-merged PR carrying the `<!-- drift-changeset: <key> -->` marker tells the workflow a human decided against that exact changeset, so it stops re-proposing it (a genuinely different drift hashes to a different key and is unaffected; a **merged** PR is an accepted decision and is never read as a rejection). A still-**open** PR carrying the marker always wins over a closed one, so closing a duplicate does not reject the changeset the surviving PR is still proposing. The suppression is **not silent, and not repetitive** — the first run after the closure posts a Slack line naming the closing PR, then records an ack marker in that PR's body so the identical line is not re-posted every morning for as long as the rejection stands (which is for ever: the closure is permanent and the changeset key is date-independent). Delete that ack marker and the next run reports the suppression again. **To un-suppress: REOPEN that PR** — it becomes the pending proposal again, and the registry stays drifted until you do. Deleting the `<!-- drift-changeset: … -->` marker from the closed PR's body does **not** un-suppress: the marker self-heal now covers closed PRs and puts it back, because that marker going missing is far more often a human rewriting the body (to write down _why_ they declined) than a deliberate un-suppression — and losing it that way used to resurrect the rejected changeset every morning, permanently. Reopening is the deliberate act; a body edit is not.

Expand Down
5 changes: 3 additions & 2 deletions scripts/drift-sync-check.ts
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,9 @@
* anti-cheat predicate (`drift-success-predicate.ts`, spec §3/§6).
*
* `drift-sync.ts` (C2) never freewrites a fix — it only ever performs one of
* two deterministic, data-only edits: (a) a zero-reference deprecation
* removal in `model-registry.ts`, or (b) drop a needs-human dedup note file
* two deterministic, data-only edits: (a) append one comment-marked family
* literal to a `model-registry.ts` set (a recorded deprecation, or a
* human-approved new family), or (b) drop a needs-human dedup note file
* under `drift-proposals/`. Because the SYNC path can no longer produce an
* arbitrary diff, verifying it is "real" no longer needs adversarial-intent
* modeling or TS-diff parsing (the predicate's whole reason for being 916
Expand Down
Loading
Loading