feat(engine): inbound observability — health endpoint, forged-token signal, log gaps (HT-44)

[zaridan]

Summary

The observability half of HT-44 — the live-threading-verification half passed on prod 2026-07-17 and is recorded on the ticket. Scope confirmed with the maintainer this session: close the two genuine log-field gaps, and make the forged-token signal + queue/watch health genuinely alertable with no alerting stack (health endpoint + status-code pivot), including migration 019.

The forged-token signal, consumed three ways (spec §6)

decideThreading has emitted forgedTokenCount since HT-38; nothing consumed it. Now:

1. Migration 019 — inbound_deliveries.forged_token_count, written by markStoredInTx in the same fenced UPDATE as the stored transition. Queryable/aggregatable, not just a log field. (Stored-transition only, by design — documented in the migration's doc comment.)
2. forged_token_detected WARN event per stored delivery carrying ≥1 forged token, with senderAddress + target conversation — text-searchable by event name, severity-filterable.
3. The health endpoint aggregates the last 24h and trips forged-token-burst at ≥5 forged-token deliveries (a code constant; threading.md §5's exact-threshold question deliberately stays open — the mechanism no longer is).

GET /api/v1/internal/health (CRON_SECRET-guarded)

Same auth/shape as the existing internal cron endpoints, one new status contract: 200 when ok, 503 when any alert trips — so any status-code poller that can send one header (UptimeRobot/Checkly free tier) is a complete alerting stack. Report sections: queue (getStats() — previously implemented but unexposed — + 24h dead-letter growth), ledger 24h outcome counts + standing dead-letter total, forged-token 24h aggregate, per-mailbox status + watch() expiry. Six alert codes, each documented in runbook Part G with a first-response playbook.

A considered-and-dropped check is documented in health.ts's module doc: a reconcile-cursor-staleness alert can't be measured honestly with the current schema (gmail_watch_state.updated_at conflates cursor and renewal writes).

Log-field gaps closed

• Append-fallback reason (deleted/not-found) — previously swallowed in writeParsedEmail, so a fallback's inbound_ingest line read threading: 'append' while pointing at a conversation the write itself created. Now on the stored outcome + log line.
• queue_drain summary line — the DrainReport (incl. staleSkipped, the drains-overlapping signal) previously existed only in a response body Vercel Cron discards. Logged for non-quiet ticks.
• History id / cursor position: already emitted at batch level by the gmail_reconcile transport events; spec §6 now says so explicitly instead of implying it rides each pipeline record. Deliberately NOT threaded through RawInboundMessage — the pipeline stays provider-agnostic (§2); streams correlate on (mailboxId, providerMessageId).

Docs

• specs/mail/inbound-ingestion.md §6 rewritten to match reality (fields, cursor ownership, the three consumers).
• specs/deploy/gmail-inbound-runbook.md: Part F gains the health smoke line; new Part G — endpoint contract, monitor wiring (incl. the CRON_SECRET-sharing tradeoff), alert-code → first-response table, structured-event reference.

Mail-semantics note (charter invariant #5)

No threading/ingest behavior changes: the append-fallback path, forged-token handling, and all decisions are untouched — this PR only records and surfaces what already happens. The one store-write change is markStoredInTx carrying one extra column in the same fenced UPDATE. All 8 observed-fixture-derived acceptance behaviors still pass through the pipeline suite.

Testing

• npm run typecheck / npm run lint / npm run test — all green by their own exit codes on the final tree: 51 files / 1070 tests (baseline 50/1055; +15).
• New: health.test.ts (PGlite + the real createPostgresQueue.getStats(), every alert exercised both tripped and quiet), health-route tests in app.test.ts (200/503 pivot, 401, 405, verbatim body), a composition drive-through in root.test.ts, migration-019 non-fresh upgrade test, forged-token pipeline test (ledger column + WARN event), append-fallback assertions on the existing fallback tests.

🤖 Generated with Claude Code

Summary by CodeRabbit

• New Features
  • Added an authenticated internal health-check endpoint (HT-44) with clear healthy/unhealthy responses, including queue, ingestion, mailbox, and Gmail watch status.
  • Enhanced monitoring signals for suspicious reply-token activity, including persistent tracking and alertable burst detection.
  • Improved reply threading behavior when prior conversations are deleted/unavailable by creating a new conversation.
• Documentation
  • Updated the deployment runbook with operator smoke-check steps and ongoing alerting guidance based on the health endpoint.
• Tests / Reliability
  • Expanded unit and integration coverage for the health endpoint, forged-token alerts, and revised ingestion fallback behavior.
• Database
  • Added a migration to persist forged-token counts on stored deliveries.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 5c275e35-c39e-4e7d-961b-610a0567e26c

📥 Commits

Reviewing files that changed from the base of the PR and between 20a4866 and 0aa78c2.

📒 Files selected for processing (1)

• specs/deploy/gmail-inbound-runbook.md

🚧 Files skipped from review as they are similar to previous changes (1)

• specs/deploy/gmail-inbound-runbook.md

---

📝 Walkthrough

Walkthrough

This change persists forged-token counts and structured ingest signals, adds an aggregated internal health report and authenticated endpoint, wires health checks into the composition root, expands coverage, and documents smoke checks, alerting, and log correlation.

Changes

Health and forged-token observability

Layer / File(s)	Summary
Persist forged-token counts src/db/migrate.ts, src/db/migrate.test.ts, src/store/inbound-deliveries.ts, src/store/inbound-deliveries.test.ts	Migration 019 adds inbound_deliveries.forged_token_count; storage APIs persist, retrieve, and test the value during the stored transition.
Emit ingest and forged-token signals src/mail/ingest.ts, src/mail/ingest.test.ts, specs/mail/inbound-ingestion.md	Ingestion reports append fallbacks, persists forged-token counts, and emits inbound_ingest and forged_token_detected events.
Compute and expose health reports src/composition/health.ts, src/composition/health.test.ts, src/composition/app.ts, src/composition/app.test.ts, src/composition/root.ts, src/composition/root.test.ts	Health checks aggregate queue, ingest, forged-token, and mailbox conditions; the authenticated endpoint returns the full report with status 200 or 503.
Document smoke checks and alerting specs/deploy/gmail-inbound-runbook.md	The runbook documents the health smoke check, HTTP monitoring, alert codes, response actions, and structured-log correlation.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Client
  participant createAppHandler
  participant runHealthCheck
  participant Database
  participant QueueAdapter
  Client->>createAppHandler: GET /api/v1/internal/health with CRON_SECRET
  createAppHandler->>runHealthCheck: execute health check
  runHealthCheck->>Database: read aggregate health metrics
  runHealthCheck->>QueueAdapter: getStats()
  runHealthCheck-->>createAppHandler: HealthReport
  createAppHandler-->>Client: report with HTTP 200 or 503

Loading

Possibly related PRs

• Helpthread/helpthread#9: Introduces the forged-token count consumed by this persistence and alerting flow.
• Helpthread/helpthread#43: Established the composition-layer cron routing extended by the health endpoint.
• Helpthread/helpthread#48: Modified the inbound-delivery lease and stored-transition plumbing extended here.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarizes the main change: inbound observability with a health endpoint, forged-token signal, and logging updates for HT-44.
Docstring Coverage	✅ Passed	Docstring coverage is 90.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feat/ht-44-inbound-observability

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@specs/deploy/gmail-inbound-runbook.md`:
- Around line 273-276: Add an appropriate language identifier, such as text or
http, to the fenced code block containing the health-check request in the
deployment runbook, preserving its existing contents.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 68a105ae-398d-441e-950b-b3c00fe4d297

📥 Commits

Reviewing files that changed from the base of the PR and between 502bc6b and 20a4866.

📒 Files selected for processing (14)

• specs/deploy/gmail-inbound-runbook.md
• specs/mail/inbound-ingestion.md
• src/composition/app.test.ts
• src/composition/app.ts
• src/composition/health.test.ts
• src/composition/health.ts
• src/composition/root.test.ts
• src/composition/root.ts
• src/db/migrate.test.ts
• src/db/migrate.ts
• src/mail/ingest.test.ts
• src/mail/ingest.ts
• src/store/inbound-deliveries.test.ts
• src/store/inbound-deliveries.ts