Skip to content

Nothing surfaces when a deployed project falls behind main #176

Description

@zaridan

Both production Vercel projects were silently far behind main, for weeks, with no signal anywhere.

Discovered 2026-08-02 while deploying HT-101 (#165).

Project Was deployed at Behind by
helpthread (engine, desk.resonantiq.app) df3431c — PR #94, passkey login 21 commits
helpthread-inbox (UI, inbox.resonantiq.app) 502bc6b — PR #71, per-Agent login ~30 commits

Why this is worse than it sounds

The UI gap meant the entire operator-facing half of HT-101 did not exist in production. The engine could accept an IMAP mailbox connection; no human could reach that screen in a browser. A feature can be merged, green, and live on the engine while remaining completely unusable, and nothing says so.

It also hid itself: because nothing reports deployed-vs-main, the only way to notice was to go looking. docs/decisions records a related observation from #74"merge push didn't trigger Vercel git build — verify next merge auto-deploys" — and the next merge evidently was not verified. The note was right and it still did not prevent the drift, because a note is not a mechanism.

Why it happens

Both projects deploy from main only, with no PR previews. Git-triggered builds are not reliably firing — every recent production deployment on the engine was source: cli, i.e. someone ran vercel deploy --prod by hand. When nobody runs it, nothing ships, and nothing complains.

What would have caught it

Roughly in order of cost:

  1. Expose the running commit. The engine already serves GET /api/v1/internal/health; adding the build's git SHA to that payload makes drift checkable in one request. The UI needs an equivalent — a /api/health or a build-stamped meta tag.
  2. Alert on drift. A scheduled check comparing each project's deployed SHA against origin/main, failing when they differ by more than N commits or M hours. This is the actual fix; everything else is diagnosis.
  3. Fix or remove the git integration. If auto-deploy on merge is supposed to work, it is broken and should be repaired. If manual deploys are deliberate, the runbook should say so plainly so nobody assumes a merge shipped.

Scope note

This is not #152. That issue is about migrations being safe and obvious. This one is about code not reaching production at all, which is a separate failure with a separate fix — and the two compound: a stale deploy plus a manual migration step means production can sit in a state nobody has ever tested.

Acceptance

  • The running commit is observable for both projects without a Vercel login.
  • Drift between deployed SHA and main raises something a human sees, on a schedule, without anyone remembering to look.
  • The runbook states whether merging deploys, in one unambiguous sentence.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions