Both production Vercel projects were silently far behind main, for weeks, with no signal anywhere.
Discovered 2026-08-02 while deploying HT-101 (#165).
| Project |
Was deployed at |
Behind by |
helpthread (engine, desk.resonantiq.app) |
df3431c — PR #94, passkey login |
21 commits |
helpthread-inbox (UI, inbox.resonantiq.app) |
502bc6b — PR #71, per-Agent login |
~30 commits |
Why this is worse than it sounds
The UI gap meant the entire operator-facing half of HT-101 did not exist in production. The engine could accept an IMAP mailbox connection; no human could reach that screen in a browser. A feature can be merged, green, and live on the engine while remaining completely unusable, and nothing says so.
It also hid itself: because nothing reports deployed-vs-main, the only way to notice was to go looking. docs/decisions records a related observation from #74 — "merge push didn't trigger Vercel git build — verify next merge auto-deploys" — and the next merge evidently was not verified. The note was right and it still did not prevent the drift, because a note is not a mechanism.
Why it happens
Both projects deploy from main only, with no PR previews. Git-triggered builds are not reliably firing — every recent production deployment on the engine was source: cli, i.e. someone ran vercel deploy --prod by hand. When nobody runs it, nothing ships, and nothing complains.
What would have caught it
Roughly in order of cost:
- Expose the running commit. The engine already serves
GET /api/v1/internal/health; adding the build's git SHA to that payload makes drift checkable in one request. The UI needs an equivalent — a /api/health or a build-stamped meta tag.
- Alert on drift. A scheduled check comparing each project's deployed SHA against
origin/main, failing when they differ by more than N commits or M hours. This is the actual fix; everything else is diagnosis.
- Fix or remove the git integration. If auto-deploy on merge is supposed to work, it is broken and should be repaired. If manual deploys are deliberate, the runbook should say so plainly so nobody assumes a merge shipped.
Scope note
This is not #152. That issue is about migrations being safe and obvious. This one is about code not reaching production at all, which is a separate failure with a separate fix — and the two compound: a stale deploy plus a manual migration step means production can sit in a state nobody has ever tested.
Acceptance
- The running commit is observable for both projects without a Vercel login.
- Drift between deployed SHA and
main raises something a human sees, on a schedule, without anyone remembering to look.
- The runbook states whether merging deploys, in one unambiguous sentence.
Both production Vercel projects were silently far behind
main, for weeks, with no signal anywhere.Discovered 2026-08-02 while deploying HT-101 (#165).
helpthread(engine,desk.resonantiq.app)df3431c— PR #94, passkey loginhelpthread-inbox(UI,inbox.resonantiq.app)502bc6b— PR #71, per-Agent loginWhy this is worse than it sounds
The UI gap meant the entire operator-facing half of HT-101 did not exist in production. The engine could accept an IMAP mailbox connection; no human could reach that screen in a browser. A feature can be merged, green, and live on the engine while remaining completely unusable, and nothing says so.
It also hid itself: because nothing reports deployed-vs-
main, the only way to notice was to go looking.docs/decisionsrecords a related observation from #74 — "merge push didn't trigger Vercel git build — verify next merge auto-deploys" — and the next merge evidently was not verified. The note was right and it still did not prevent the drift, because a note is not a mechanism.Why it happens
Both projects deploy from
mainonly, with no PR previews. Git-triggered builds are not reliably firing — every recent production deployment on the engine wassource: cli, i.e. someone ranvercel deploy --prodby hand. When nobody runs it, nothing ships, and nothing complains.What would have caught it
Roughly in order of cost:
GET /api/v1/internal/health; adding the build's git SHA to that payload makes drift checkable in one request. The UI needs an equivalent — a/api/healthor a build-stamped meta tag.origin/main, failing when they differ by more than N commits or M hours. This is the actual fix; everything else is diagnosis.Scope note
This is not #152. That issue is about migrations being safe and obvious. This one is about code not reaching production at all, which is a separate failure with a separate fix — and the two compound: a stale deploy plus a manual migration step means production can sit in a state nobody has ever tested.
Acceptance
mainraises something a human sees, on a schedule, without anyone remembering to look.