From 6e29f273491f9686f2438a582fc3274dd4fe9add Mon Sep 17 00:00:00 2001 From: TJ Baker <1617679+zaridan@users.noreply.github.com> Date: Mon, 20 Jul 2026 10:19:25 -0700 Subject: [PATCH 1/5] docs(charter): scheduled fetch as first-class intake + mailbox connection spec (HT-92) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit §2's "no daemons, no polling loops" is restated as "no daemons, no long-running processes." Inbound mail may now arrive either by push webhook or by a bounded, stateless scheduled fetch — connect, read from a stored cursor, exit. Push stays preferred where a provider offers it; scheduled fetch becomes a first-class primary transport rather than only a reconciliation backstop. The original wording conflated a daemon (stays resident, holds a connection — what makes FreeScout need a persistent PHP host) with a scheduled fetch (an invocation that ends). Only the first is incompatible with serverless. §4 already permitted "bounded reconciliation fetches, never a long-running poller," and the daily watch-maintenance cron has always been exactly that: it enqueues the same reconcile job the push path enqueues, and that job reads the mailbox's stored cursor, never the push notification's historyId. The engine already did on a schedule what §2 appeared to forbid. Verified live against the dogfood desk before amending: - IMAP + app password authenticated help@resonantiq.app, returned raw RFC822 via BODY.PEEK[], and a real reply's References chain carried the ht.ht1.… signed token intact — the §2 threading anchor holds. - SMTP + the same credential sent with our own Message-ID preserved verbatim, confirmed by rfc822msgid: lookup at the recipient. Unlocks dropping the Pub/Sub dependency (six setup steps, including the two that fail silently) and IMAP/SMTP connection by app password — no DNS, no MX change, no cloud console, no third-party account. Consequential wording updated at the opening summary, the FreeScout comparison, the serverless positioning line, and §4's Phase 1. Adds specs/mail/mailbox-connection.md: transport behaviour, provider support matrix (Microsoft 365 requires OAuth — app passwords are unreachable there), credential handling, the connect screen's behavioural contract, and build order. Screen pixels are deferred to a Claude Design session per the UI fidelity rule. Co-Authored-By: Claude Opus 4.8 --- CHARTER.md | 22 ++++-- specs/mail/mailbox-connection.md | 121 +++++++++++++++++++++++++++++++ 2 files changed, 138 insertions(+), 5 deletions(-) create mode 100644 specs/mail/mailbox-connection.md diff --git a/CHARTER.md b/CHARTER.md index 146b585..0ccdc31 100644 --- a/CHARTER.md +++ b/CHARTER.md @@ -1,21 +1,21 @@ # Helpthread — Founding Charter -> Helpthread is an open-source, serverless helpdesk — shared inbox, threaded email conversations, knowledge base — built for teams who live on Vercel and Supabase rather than a LAMP server. It is aimed at the bar Help Scout set for ease of use — the experience this project exists to make ownable — rebuilt in modern TypeScript with no daemons, no polling loops, and an extension system designed for how serverless software actually deploys. FreeScout, the established self-hosted PHP helpdesk, proved the self-hosted market is real; Helpthread is built for that market on modern rails. Core is AGPL-3.0; the project is dogfooded first, built by the team behind Resonant IQ as their own production support system before anyone else touches it. +> Helpthread is an open-source, serverless helpdesk — shared inbox, threaded email conversations, knowledge base — built for teams who live on Vercel and Supabase rather than a LAMP server. It is aimed at the bar Help Scout set for ease of use — the experience this project exists to make ownable — rebuilt in modern TypeScript with no daemons, no long-running processes, and an extension system designed for how serverless software actually deploys. FreeScout, the established self-hosted PHP helpdesk, proved the self-hosted market is real; Helpthread is built for that market on modern rails. Core is AGPL-3.0; the project is dogfooded first, built by the team behind Resonant IQ as their own production support system before anyone else touches it. ## 1. Mission & positioning Helpthread starts from a personal fact: its founder ran customer support on Help Scout for years and loved it. Help Scout remains the gold standard for a helpdesk that feels effortless on both sides of the conversation — and that experience is the bar this project holds itself to. What Help Scout never offered is ownership: your helpdesk, your data, your customization, on your own infrastructure, with no rent forever. Help Scout's ease, fully owned — that combination is the product. -The self-hosted market is real: thousands of teams run FreeScout instead of paying rent to Zendesk or Help Scout. We ran it ourselves — and hit its walls. It's a decade-old Laravel 5.5 app held together by IMAP polling daemons, its extensibility means dropping PHP into a runtime plugin folder, and even basic customization — the knowledge base's design, its URLs — is out of reach without forking. Proof of the demand, and proof of the ceiling. FreeScout serves one further purpose here: because Help Scout is closed SaaS you can't inspect, FreeScout is our open *window* into that experience — a self-hostable reference for modeling the interface toward Help Scout's ease of use. It is a UX reference, never a source of code. The core is our own, built on modern, permissively-licensed foundations. +The self-hosted market is real: thousands of teams run FreeScout instead of paying rent to Zendesk or Help Scout. We ran it ourselves — and hit its walls. It's a decade-old Laravel 5.5 app that needs a persistent host running scheduler daemons, its extensibility means dropping PHP into a runtime plugin folder, and even basic customization — the knowledge base's design, its URLs — is out of reach without forking. Proof of the demand, and proof of the ceiling. FreeScout serves one further purpose here: because Help Scout is closed SaaS you can't inspect, FreeScout is our open *window* into that experience — a self-hostable reference for modeling the interface toward Help Scout's ease of use. It is a UX reference, never a source of code. The core is our own, built on modern, permissively-licensed foundations. -Helpthread is the answer for people who already live on serverless infrastructure. Same ownership promise — rebuilt for a world of edge functions, managed Postgres, and push-based delivery instead of cron loops watching a mailbox. Two audiences have to come out of this feeling the same way: customers should find getting support dead easy, and operators should find running the thing dead easy. Neither of those is negotiable in favor of the other. +Helpthread is the answer for people who already live on serverless infrastructure. Same ownership promise — rebuilt for a world of edge functions, managed Postgres, and push-based delivery where providers offer it, bounded scheduled fetches where they don't — and no resident process either way. Two audiences have to come out of this feeling the same way: customers should find getting support dead easy, and operators should find running the thing dead easy. Neither of those is negotiable in favor of the other. ## 2. Product principles - **Innovate on the platform. Be boringly faithful on mail semantics.** Everything about *how* Helpthread runs — compute model, storage, deployment — is fair game for rethinking from scratch. How it parses, threads, and sends email is not. Mature systems — FreeScout and Chatwoot among them — have earned a decade of edge-case scars in production email handling; we respect that earned knowledge, reproduce proven behavior, and verify against fixtures rather than re-deriving from first principles. Any change to mail behavior needs proof of equivalence or an explicit, written justification — not a hunch that the old way looked wrong. During this project's own early development we watched well-intentioned "improvements" to mail handling silently destroy message content in testing; that lesson is why this rule exists and why it's not up for debate per-PR. - **Threading authority lives on the outbound side.** Inbound threading headers (`References`, `In-Reply-To`) are written by every mail client on earth, inconsistently — they cannot be trusted. The one header the engine fully controls is the Message-ID it emits. So threading is anchored there: signed reply tokens in outbound Message-IDs, authenticating replies on their way back in — a pattern as old as mailing-list software. This is the mechanism the whole system leans on; treat it accordingly. - **API-first — and the UI isn't the only client.** Anything the UI can do, a typed public API can do. The UI is a client of that API, not a special case; so are AI agents. An MCP server exposing the same operations ships as a first-class client of the same API, so any operator can point their own AI tooling at their own helpdesk. -- **Serverless-native, Vercel-first, not Vercel-only.** No daemons, no polling loops. Inbound mail arrives via push webhooks (Gmail push through Pub/Sub, and equivalents as we add providers); anything that needs to happen later is a scheduled action or a cron trigger, not a process sitting in a loop. The engine's core reaches every platform service through interfaces the project owns — see the platform posture note in the architecture section. +- **Serverless-native, Vercel-first, not Vercel-only.** No daemons, no long-running processes. Inbound mail arrives either by push webhook (Gmail push through Pub/Sub, and equivalents as we add providers) or by a bounded, stateless scheduled fetch — a function that connects, reads from a stored cursor, and exits. Push is preferred wherever a provider offers it, on latency grounds; scheduled fetch is a first-class primary transport, not merely a reconciliation backstop. Nothing stays resident either way: no held connections, no IMAP IDLE, no worker loops. Anything that needs to happen later is a scheduled action or a cron trigger, not a process sitting in a loop. The engine's core reaches every platform service through interfaces the project owns — see the platform posture note in the architecture section. *(Amended 2026-07-20 — scheduled-fetch intake; see §7 appendix.)* - **Own your data.** Self-hosted on the operator's own Vercel and Supabase accounts. Conversation data never touches Resonant IQ-operated infrastructure — we don't sit in the data path — and AI features preserve that: assistants call model providers the operator configures, with the operator's own keys. **The promise is absolute and carries no exception.** *(The 2026-07-19 "managed hosting" carve-out was **rescinded 2026-07-20** on TJ's review — he had selected managed hosting from a picker that labelled it "Recommended", and rejected the model once its consequences were clear. See the §7 appendix.)* ## 3. Licensing & intellectual property @@ -63,7 +63,7 @@ The founding public API surface is six conversation operations: list conversatio No dates; phases are ordered by dependency, not calendar. - **Phase 0 — Foundations.** Name, domains, GitHub org, and npm org secured (done). This charter. A public-ready engine repo with clean history. -- **Phase 1 — Core engine, dogfooded.** Mail engine: event-driven ingestion (bounded reconciliation fetches, never a long-running poller), parsing, threading, sending, signed reply tokens, auto-responder handling, bounce handling, HTML sanitization. The six-operation conversation API. An agent inbox UI. Gmail push for inbound. This runs as Resonant IQ's actual production support desk before it runs as anyone else's. +- **Phase 1 — Core engine, dogfooded.** Mail engine: event-driven ingestion (bounded reconciliation and intake fetches, never a long-running poller), parsing, threading, sending, signed reply tokens, auto-responder handling, bounce handling, HTML sanitization. The six-operation conversation API. An agent inbox UI. Gmail push for inbound. This runs as Resonant IQ's actual production support desk before it runs as anyone else's. - **Phase 2 — Production cutover.** Resonant IQ retires its FreeScout instance and switches to Helpthread at the config level — a clean cutover with no legacy data requiring migration. - **Phase 3 — Public launch.** Deploy-to-Vercel button, public docs site, and the start of a community — **with the marketplace live on day one**: license keys, module distribution, and the first paid modules (the knowledge base and AI-powered modules — draft-reply suggestions, auto-triage, KB-grounded auto-answers in the widget — lead the catalog), all having served as Resonant IQ's own install path through the dogfood phase. A helpdesk launched free with its commercial layer already real avoids the worse story: a community formed around ambiguity about how the project sustains itself. @@ -99,4 +99,16 @@ Solo-maintainer, BDFL model for now — there is one project, one person account ~~§2 — the "own your data" bullet's promise is stated as *conversation data never touches Resonant IQ-operated infrastructure — we don't sit in the data path*, with one explicit, opt-in exception: a module an operator enrolls in Resonant IQ's managed hosting processes that operator's conversation data (and holds the operator-supplied keys it needs) on Resonant IQ infrastructure, under disclosed data-handling terms consented to at enrollment. This also corrects a mis-scoped earlier phrasing that put "the core" out of the data path — the operator-run core *is* the data path; the promise that matters is that **Resonant IQ-operated** infrastructure stays out of it. Rationale: the managed-hosting amendment (same date, above) made an unqualified promise falsifiable for enrolled operators; a data promise that is precisely true beats one that is absolutely worded and false in an opt-in case. The mail engine and system of record remain on the operator's own accounts in every configuration, and the self-host path preserves the original absolute promise. Decided by TJ acting as counsel, 2026-07-19; the managed-hosting data-handling terms themselves remain on the §8 pre-revenue counsel gate (`specs/modules/marketplace-v1.md` §8).~~ +**Amended 2026-07-20 (HT-92, scheduled-fetch intake):** §2 — the "Serverless-native" bullet's prohibition is restated from *"no daemons, no polling loops"* to **"no daemons, no long-running processes."** Inbound mail may arrive either by push webhook or by a **bounded, stateless scheduled fetch** — a function that connects, fetches from a stored cursor, and exits. Push remains preferred wherever a provider offers it, on latency grounds, but scheduled fetch is now a first-class primary transport rather than only a reconciliation backstop. Consequential wording updated at §1's opening summary, the FreeScout comparison, the serverless positioning line, and §4's Phase 1 description. + +Rationale: the original wording conflated two separable things — a *daemon* (a process that stays resident and holds an open connection, which is what makes FreeScout require a persistent PHP host) and a *scheduled fetch* (an invocation that ends). Only the first is incompatible with serverless. §4's Phase 1 language already permitted "bounded reconciliation fetches, never a long-running poller," and the daily watch-maintenance cron has always been exactly that — it enqueues the same reconcile job the push path enqueues, and that job reads the mailbox's *stored cursor*, never the push notification's `historyId`. The engine already did on a schedule what §2 appeared to forbid; this amendment removes an internal inconsistency rather than introducing a capability. + +What it unlocks: dropping the Pub/Sub dependency (six setup steps, including the two that fail silently — the domain-restricted-sharing org-policy block and the missing `serviceAccountTokenCreator` grant), and IMAP/SMTP mailbox connection via app password, which needs no DNS records, no MX change, no cloud console, and no third-party provider account. Measured against the ~26-step, 4-console setup an adopter faces today (HT-82 dogfood), this is the difference between Helpthread being evaluable and not being evaluated. §1's own standard — *operators should find running the thing dead easy* — was not being met. + +Verified before amending (2026-07-20, live against the dogfood desk): IMAP + app password authenticated `help@resonantiq.app` and returned raw RFC822 with a real reply's `ht.ht1.…` signed token intact in its `References` chain, connection closed on exit; SMTP + the same credential sent with our own `Message-ID` preserved verbatim through Gmail's submission path. + +What does not change: no process may stay resident — no IMAP IDLE, no held connections, no worker loops; each fetch is a fresh bounded invocation. Threading authority stays on the outbound Message-ID (§2). Mail semantics remain sacred (§2) — any new transport ships with fixtures proving equivalence before it becomes a default. The data-path promise (§2, "Own your data") is untouched, and is strengthened: a direct mailbox connection puts fewer third parties in the path than a provider webhook would. + +Positioning consequence, accepted deliberately: §1 previously sold against FreeScout partly on this line. The honest differentiator survives and is restated — FreeScout requires a host running resident scheduler daemons; Helpthread runs the same logical work as stateless invocations on infrastructure the operator does not administer. The claim was never truthfully "we never fetch on a schedule," since the engine always did. It is "nothing of ours stays running." Decided by TJ, 2026-07-20. + **Amended 2026-07-19 (HT-75):** §3 and §4 — the knowledge base reclassified from free-forever core to a paid first-party module. FreeScout's own reference instance runs its Knowledge Base as a paid purchase (`specs/ui/admin-ia.md`); keeping ours core was chosen 2026-07-18 as a marketed free-vs-FreeScout differentiator (`specs/modules/catalog.md` §1), and TJ reversed that call on review. Not a retroactive paywall under the §3 invariant — the knowledge base was never shipped (`specs/modules/catalog.md` §2.1's shipped list never included it). Full paid-module catalog: `specs/modules/catalog.md`. diff --git a/specs/mail/mailbox-connection.md b/specs/mail/mailbox-connection.md new file mode 100644 index 0000000..db7df42 --- /dev/null +++ b/specs/mail/mailbox-connection.md @@ -0,0 +1,121 @@ +# Mailbox connection — scheduled-fetch intake, IMAP/SMTP transport, and the connect screen + +**Ticket:** HT-92 · **Status:** spec, implementation not started +**Charter dependency:** the scheduled-fetch amendment (CHARTER.md §7, 2026-07-20). This spec is unimplementable without it. + +## 1. The problem + +Getting email working on a fresh install takes ~26 steps across four consoles — 4–6 hours for an operator who knows Google Cloud, longer for one who doesn't. About half of those steps exist solely to make Gmail **push** work, and two of them fail silently: + +- the domain-restricted-sharing org policy blocks the Pub/Sub IAM grant with no useful error +- a CLI-created push subscription needs an extra `roles/iam.serviceAccountTokenCreator` grant, without which push simply never arrives + +There is also no UI for connecting a mailbox. The operator runs a raw `curl` against `POST /api/v1/inbound/gmail/connect` to obtain a consent URL (`specs/deploy/gmail-inbound-runbook.md`). + +**Success criterion, stated as a test:** a new operator with an existing mailbox goes from deployed app to first ingested email in under five minutes, touching no DNS records, no cloud console, and no third-party signup. + +## 2. What the spike established (2026-07-20, verified live) + +**Scheduled fetch already ships.** `runGmailWatchMaintenance` (`src/mail/gmail-watch-maintenance.ts`, the daily cron in `vercel.json`) performs "a bounded reconciliation sweep" that enqueues *the same reconcile job the push path enqueues*. That job (`src/mail/gmail-reconcile.ts`) reads the mailbox's **stored cursor** — explicitly never the push notification's `historyId` — then calls `history.list` followed by `messages.get?format=raw`. It consumes nothing from the push payload. + +Consequence: **push only makes the same job run sooner.** Making scheduled fetch the primary intake is a scheduling change, not new fetch code. The lease around `history.list` (`claimReconcileLease`) already prevents concurrent runs from double-fetching. + +**IMAP + app password works.** Probed `help@resonantiq.app` directly: authenticated, `SELECT INBOX` (`UIDVALIDITY=1`, `UIDNEXT=32`, `EXISTS=29`), `UID FETCH BODY.PEEK[]` returned raw RFC822 for five messages in 1174 ms, connection closed on exit. A real reply's `References` chain carried the `ht.ht1.…` signed reply token intact — the CHARTER.md §2 threading anchor holds on this transport. + +**SMTP + app password works.** Same credential, `smtp.gmail.com:465`, `AUTH PLAIN`, 1833 ms. Our own `Message-ID` was preserved verbatim, confirmed by an `rfc822msgid:` lookup at the recipient. + +Worth recording: HT-49 exists because Gmail's `users.messages.send` **API** rewrites the `Message-ID` we set, which is why the reply token had to move into `References` (`src/mail/send.ts`). SMTP submission did not rewrite it. One observation is not proof of a general rule, and the `References` mechanism works on both transports — so this is a point in SMTP's favour, not a reason to change the threading model. + +**Outbound needs no DNS.** Because replies go through the operator's own mail server, they are signed by that server's DKIM, from its IPs, on a domain whose SPF already authorises it. Every provider-webhook alternative considered (Postmark, Resend, SES, Cloudflare) would reintroduce SPF/DKIM/DMARC setup. This transport does not. + +## 3. Provider support + +Verified 2026-07-20. App-password availability gates the whole approach. + +| Provider | App password | Notes | +|---|---|---| +| Gmail consumer | Yes | 2SV required; unavailable under Advanced Protection | +| Google Workspace | Yes | Unless admin blocks it or enforces security-key-only 2SV | +| Fastmail | Yes | The only IMAP method offered | +| Zoho / Yahoo / iCloud | Yes | 2FA required | +| cPanel / self-hosted | Yes | The mailbox password is the credential | +| **Microsoft 365 business** | **No** | Basic auth removed and unre-enablable — **OAuth mandatory** | +| Outlook.com consumer | Unresolved | Microsoft's own docs contradict; test before claiming support | + +Two consequences: + +1. **OAuth is a required second connector, not a nice-to-have** — Microsoft business mail is unreachable without it. The HT-40 OAuth machinery is reused, not rebuilt. +2. **Generic IMAP still earns its place** — it is what makes Helpthread work with any mailbox rather than only Google and Microsoft. + +Watch item: Google documents app passwords as legacy with no committed lifetime. No announced EOL, but no commitment either; OAuth is the hedge. + +**Recommended default per provider:** OAuth for Google and Microsoft (scoped, revocable, no security warning); app password everywhere else. + +## 4. The connection screen + +**Design authority.** Per `CLAUDE.md`, the Agent Inbox UI's pixel source of truth is the Claude Design prototype, and deviation requires explicit sign-off. This section defines **behaviour and contract only** — the visual design needs a design-project session before implementation. An engineer must not invent this layout. + +Behavioural contract: + +- **Provider presets.** Recognise the address's domain and prefill IMAP/SMTP hosts and ports. Unknown domains expand an Advanced section. +- **Method availability is provider-dependent.** For an M365 business domain the app-password option is disabled *with an explanation*, never silently absent. +- **`Check connection`** performs a real IMAP login + `SELECT INBOX` and a real SMTP handshake + `AUTH`, reporting each leg independently. It must never report success from a config-shape check alone. +- **`Send test email`** sends an actual message through the operator's SMTP and confirms it is observed back through the fetch path. This round trip is the single most valuable element on the screen, and it is what today's setup path has no equivalent of — which is why silent misconfiguration currently survives to production. +- **Errors surface verbatim** — the provider's actual IMAP/SMTP rejection text, plus a mapped hint for the common cases (2SV not enabled, app passwords blocked by admin policy, wrong port, mailbox not found). Never a bare "connection failed." + +## 5. Transport behaviour + +### Inbound — bounded scheduled fetch + +A cron entry alongside the existing four in `vercel.json`. Each invocation: connect, `SELECT INBOX`, fetch messages above the stored UID cursor, hand raw bytes to the pipeline, persist the new cursor, disconnect. + +- **No IDLE, no held connections, no resident process.** The connection opens and closes within the invocation. This is the charter constraint, and it must not be optimised away for latency later. +- **Track `UIDVALIDITY` alongside the UID cursor.** If the server changes `UIDVALIDITY`, every stored UID is meaningless and the cursor must be rebuilt. Skipping this silently drops or re-ingests mail. +- **Bound the batch.** `maxDuration` is 50 s (`vercel.json`). Cap messages per invocation and continue on the next tick rather than risk a timeout mid-fetch. +- **Fetch raw.** `FETCH BODY.PEEK[]` — full RFC822, `.PEEK` so `\Seen` is not set. This satisfies `src/providers/inbound-email.ts` natively: raw bytes in, parsed exactly once by `parseInboundEmail`. No IMAP library's convenience parser may touch the message. + +### Outbound — the operator's own SMTP + +Stated explicitly in the operator docs, because it is the quiet advantage: replies are sent through the mail server the operator already runs, so deliverability is already correct and requires no DNS work. + +### Credentials + +App passwords are long-lived secrets granting full mailbox access. They must be encrypted at rest via the existing `HELPTHREAD_TOKEN_ENC_KEY` path (`src/store/token-crypto.ts`, AES-256-GCM) already used for OAuth tokens, never logged, never returned by any API read, and write-only in the UI — show a "configured" state, never the value. + +Note the asymmetry worth telling operators about: an app password cannot be scoped and does not expire; an OAuth token is scoped to `gmail.readonly` + `gmail.send` and is revocable from the account. + +### Operational + +**Connection churn is the risk, not fetch frequency.** No provider publishes a per-minute polling limit, but reconnecting every 60 s while the operator's own devices also hold connections is where limits bite. Gmail's documented ceiling is bandwidth (2,500 MB/day download), which only matters on large-attachment backfill. Make the interval configurable and default it conservatively. + +## 6. Addressing — mail that arrives under another address + +Operators commonly route `support@company.com` into the mailbox they connect, rather than connecting `support@` itself. This is entirely the operator's own plumbing — a forwarding rule, an alias, a group — and the engine cannot and should not distinguish it from directly-addressed mail. Both are just messages in the mailbox. + +It does have one design consequence: such mail carries `To: support@company.com`, not the connected mailbox address. **The mailbox therefore needs a configured list of addresses the desk answers for**, or the engine may not recognise that mail as belonging to it. + +That is a configuration detail on the mailbox record, not a transport concern. + +## 7. Build order + +1. **Drop the Pub/Sub dependency** — reschedule the existing reconcile sweep. Least new code, largest reduction in setup steps, removes both silent-failure traps and the billing requirement (Pub/Sub is what forced it). +2. **IMAP provider adapter** behind the existing `InboundEmailProvider` seam, with fixtures proving equivalence against the Gmail path. +3. **Migrations** — UID cursor + `UIDVALIDITY`; encrypted mailbox credential. +4. **SMTP sender** behind the existing `EmailSender` seam. +5. **The connection screen** — design session first, then implementation. +6. **`Check connection` / `Send test email`**, including the round-trip verification. +7. **Microsoft OAuth connector**, reusing the HT-40 machinery. +8. Demote Gmail push in the docs to the advanced low-latency option; keep it fully supported. + +## 8. Out of scope — tracked separately + +Roughly half the current setup burden is not email at all: Supabase project creation, 26 manual migrations, two separate Vercel projects with duplicated env vars, and four hand-minted `openssl` secrets. + +The **Vercel Pro requirement** was assessed and dismissed as a blocker: three `*/1` crons already force it today, independent of mail intake, and Vercel's Hobby plan prohibits commercial use — so any company running Helpthread as a real support desk needs Pro by licence terms regardless. Evaluation friction is answered by managed hosting (CHARTER.md §7, HT-79), not by architecture. + +Also found while mapping the current path, each needing its own ticket: + +- **RLS is disabled on all 19 tables**, including `conversations` and `mailbox_oauth_tokens`. Exposure depends on whether the anon key is distributed; enabling RLS without policies would break the engine's own access, so this needs deliberate policy design. +- **Three doc-drift defects** in `specs/deploy/gmail-inbound-runbook.md`: it names an `api/[...path].ts` entrypoint that was tried and abandoned (the real file is `api/index.ts`, which documents why), says "three cron jobs" where `vercel.json` declares four, and omits `HELPTHREAD_UI_BASE_URL` and the entire web-project env set — so an operator following only the runbook gets a working engine and a non-functional UI. +- **No `.env.example`** anywhere in the repo; the 17-variable contract exists only as a prose table and as validation logic in `src/composition/config.ts`. +- **History-cursor expiry fallback** not audited. Gmail expires history cursors; the 404 path exists but has not been reviewed for a mailbox that goes quiet for an extended period. From 59cf4a84786c3318fc0d509b61768e17de5deb5c Mon Sep 17 00:00:00 2001 From: TJ Baker <1617679+zaridan@users.noreply.github.com> Date: Mon, 20 Jul 2026 13:18:09 -0700 Subject: [PATCH 2/5] docs: close adversarial-review findings on the scheduled-fetch amendment (HT-92) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three independent reviews (two Opus-tier, one Codex) attacked this PR. The amendment's central claim survived: the daily cron genuinely already enqueued the same reconcile job on the same topic, and the consumer reads the stored cursor, so this removes an internal inconsistency rather than introducing a capability. The findings below are the ones that landed. **The repo contradicted the charter.** The amendment updated CHARTER.md and nothing else, leaving "no daemons, no polling loops" quoted as binding authority in four other places — including README.md, the public front door, which directly contradicted the charter it links to. Also corrected in src/providers/inbound-email.ts, src/providers/queue.ts, and specs/modules/catalog.md, whose "no IMAP polling" rationale for dropping a FreeScout feature is now void and flagged for re-deciding on its merits. **The spec is demoted to DRAFT.** Review found three gaps in specs/mail/mailbox-connection.md, all sitting on the mail-semantics invariant, all of which block its own build order: 1. Self-echo suppression has no mechanism on SMTP+IMAP. preSuppressOwnSend pre-seeds the ledger with the send's providerMessageId — "the SAME id the transport will later report." Gmail's API supplies that shared id; SMTP submission returns no id a later IMAP FETCH will report. Without an answer, every agent reply is re-ingested as an inbound customer message. HT-49/HT-50 were this class of bug and were found live. 2. providerMessageId is undefined for IMAP. The ledger's idempotency key requires a stable per-transport id; the spec itself notes a UIDVALIDITY change makes every stored UID meaningless, so a UID-keyed ledger re-ingests the whole mailbox on reset. 3. InboundEmailProvider is webhook-shaped — both methods take an HTTP Request, which a cron-driven fetch does not have. The claims "least new code" and "behind the existing seam" rested on this and were wrong. These are recorded in the spec rather than papered over. §2's spike results and §3's provider matrix are independently verified and stand. **Smaller corrections.** "It consumes nothing from the push payload" was literally false — it consumes mailboxId; the payload's historyId is logged and never acted on. Noted too that the field carries semantically opposite values depending on trigger, which is a latent trap. The amendment was also inserted mid-appendix, breaking a strictly chronological run; moved to the end after HT-75. Verified: tsc exit 0, biome exit 0, 1512 tests pass across 75 files. Co-Authored-By: Claude Opus 4.8 --- CHARTER.md | 5 ++-- README.md | 2 +- specs/mail/mailbox-connection.md | 50 ++++++++++++++++++++++++++++++-- specs/modules/catalog.md | 5 +++- src/providers/inbound-email.ts | 22 ++++++++++---- src/providers/queue.ts | 6 ++-- 6 files changed, 76 insertions(+), 14 deletions(-) diff --git a/CHARTER.md b/CHARTER.md index 0ccdc31..3ff895c 100644 --- a/CHARTER.md +++ b/CHARTER.md @@ -99,6 +99,9 @@ Solo-maintainer, BDFL model for now — there is one project, one person account ~~§2 — the "own your data" bullet's promise is stated as *conversation data never touches Resonant IQ-operated infrastructure — we don't sit in the data path*, with one explicit, opt-in exception: a module an operator enrolls in Resonant IQ's managed hosting processes that operator's conversation data (and holds the operator-supplied keys it needs) on Resonant IQ infrastructure, under disclosed data-handling terms consented to at enrollment. This also corrects a mis-scoped earlier phrasing that put "the core" out of the data path — the operator-run core *is* the data path; the promise that matters is that **Resonant IQ-operated** infrastructure stays out of it. Rationale: the managed-hosting amendment (same date, above) made an unqualified promise falsifiable for enrolled operators; a data promise that is precisely true beats one that is absolutely worded and false in an opt-in case. The mail engine and system of record remain on the operator's own accounts in every configuration, and the self-host path preserves the original absolute promise. Decided by TJ acting as counsel, 2026-07-19; the managed-hosting data-handling terms themselves remain on the §8 pre-revenue counsel gate (`specs/modules/marketplace-v1.md` §8).~~ + +**Amended 2026-07-19 (HT-75):** §3 and §4 — the knowledge base reclassified from free-forever core to a paid first-party module. FreeScout's own reference instance runs its Knowledge Base as a paid purchase (`specs/ui/admin-ia.md`); keeping ours core was chosen 2026-07-18 as a marketed free-vs-FreeScout differentiator (`specs/modules/catalog.md` §1), and TJ reversed that call on review. Not a retroactive paywall under the §3 invariant — the knowledge base was never shipped (`specs/modules/catalog.md` §2.1's shipped list never included it). Full paid-module catalog: `specs/modules/catalog.md`. + **Amended 2026-07-20 (HT-92, scheduled-fetch intake):** §2 — the "Serverless-native" bullet's prohibition is restated from *"no daemons, no polling loops"* to **"no daemons, no long-running processes."** Inbound mail may arrive either by push webhook or by a **bounded, stateless scheduled fetch** — a function that connects, fetches from a stored cursor, and exits. Push remains preferred wherever a provider offers it, on latency grounds, but scheduled fetch is now a first-class primary transport rather than only a reconciliation backstop. Consequential wording updated at §1's opening summary, the FreeScout comparison, the serverless positioning line, and §4's Phase 1 description. Rationale: the original wording conflated two separable things — a *daemon* (a process that stays resident and holds an open connection, which is what makes FreeScout require a persistent PHP host) and a *scheduled fetch* (an invocation that ends). Only the first is incompatible with serverless. §4's Phase 1 language already permitted "bounded reconciliation fetches, never a long-running poller," and the daily watch-maintenance cron has always been exactly that — it enqueues the same reconcile job the push path enqueues, and that job reads the mailbox's *stored cursor*, never the push notification's `historyId`. The engine already did on a schedule what §2 appeared to forbid; this amendment removes an internal inconsistency rather than introducing a capability. @@ -110,5 +113,3 @@ Verified before amending (2026-07-20, live against the dogfood desk): IMAP + app What does not change: no process may stay resident — no IMAP IDLE, no held connections, no worker loops; each fetch is a fresh bounded invocation. Threading authority stays on the outbound Message-ID (§2). Mail semantics remain sacred (§2) — any new transport ships with fixtures proving equivalence before it becomes a default. The data-path promise (§2, "Own your data") is untouched, and is strengthened: a direct mailbox connection puts fewer third parties in the path than a provider webhook would. Positioning consequence, accepted deliberately: §1 previously sold against FreeScout partly on this line. The honest differentiator survives and is restated — FreeScout requires a host running resident scheduler daemons; Helpthread runs the same logical work as stateless invocations on infrastructure the operator does not administer. The claim was never truthfully "we never fetch on a schedule," since the engine always did. It is "nothing of ours stays running." Decided by TJ, 2026-07-20. - -**Amended 2026-07-19 (HT-75):** §3 and §4 — the knowledge base reclassified from free-forever core to a paid first-party module. FreeScout's own reference instance runs its Knowledge Base as a paid purchase (`specs/ui/admin-ia.md`); keeping ours core was chosen 2026-07-18 as a marketed free-vs-FreeScout differentiator (`specs/modules/catalog.md` §1), and TJ reversed that call on review. Not a retroactive paywall under the §3 invariant — the knowledge base was never shipped (`specs/modules/catalog.md` §2.1's shipped list never included it). Full paid-module catalog: `specs/modules/catalog.md`. diff --git a/README.md b/README.md index 67b3380..da6b62f 100644 --- a/README.md +++ b/README.md @@ -13,7 +13,7 @@ Open-source, serverless helpdesk — shared inbox, threaded email conversations, ## Architecture in one breath -TypeScript on Vercel (Fluid Compute, Workflows, Queues, Cron — no daemons, no polling) plus Supabase (Postgres, Auth, Storage, Realtime). Inbound mail arrives by push webhook, not IMAP polling. Threading authority lives on the outbound side: signed reply tokens in the Message-IDs the engine emits. See the charter for the reasoning. +TypeScript on Vercel (Fluid Compute, Workflows, Queues, Cron — no daemons, nothing stays resident) plus Supabase (Postgres, Auth, Storage, Realtime). Inbound mail arrives by push webhook where the provider offers one, or by a bounded scheduled fetch where it doesn't — either way as an invocation that ends, never a process sitting in a loop. Threading authority lives on the outbound side: signed reply tokens in the Message-IDs the engine emits. See the charter for the reasoning. ## Development diff --git a/specs/mail/mailbox-connection.md b/specs/mail/mailbox-connection.md index db7df42..18551f6 100644 --- a/specs/mail/mailbox-connection.md +++ b/specs/mail/mailbox-connection.md @@ -1,7 +1,49 @@ # Mailbox connection — scheduled-fetch intake, IMAP/SMTP transport, and the connect screen -**Ticket:** HT-92 · **Status:** spec, implementation not started -**Charter dependency:** the scheduled-fetch amendment (CHARTER.md §7, 2026-07-20). This spec is unimplementable without it. +**Ticket:** HT-92 · **Status: DRAFT — NOT approved to build against.** +**Charter dependency:** the scheduled-fetch amendment (CHARTER.md §7, 2026-07-20). + +> ## ⛔ Three unresolved questions block the build order in §7 +> +> Adversarial review (2026-07-20) found three gaps, all of which sit on the +> **mail-semantics invariant** (CHARTER.md §2) and therefore need answers — +> with fixtures — before any of §7 starts. They are recorded here rather than +> papered over. +> +> **1. Self-echo suppression has no mechanism on SMTP+IMAP.** +> `src/store/inbound-deliveries.ts`'s `preSuppressOwnSend` pre-seeds the +> delivery ledger with the send's `providerMessageId` — "the SAME id the +> transport will later report for that exact message." Gmail's API supplies +> that shared id. **SMTP submission returns no id a later IMAP `FETCH` will +> report.** Without an answer, every agent reply lands back in the connected +> mailbox and is ingested as a new inbound customer message. HT-49/HT-50 were +> this class of bug and were found *live*. +> Candidate answer to evaluate: suppress on our own minted `Message-ID`, which +> we control on the outbound side (CHARTER.md §2's threading authority) and +> which SMTP submission was observed to preserve verbatim — but "observed +> once" is not "proven," and this needs a fixture. +> +> **2. `providerMessageId` is undefined for IMAP.** +> `specs/mail/inbound-ingestion.md` makes `(mailboxId, providerMessageId)` the +> unique-constrained idempotency key and requires `providerMessageId` to be +> "the transport's own **stable** id." §5 of this document states that a +> `UIDVALIDITY` change makes every stored UID meaningless — so a UID-keyed +> ledger re-ingests the entire mailbox on a `UIDVALIDITY` reset, the exact +> silent duplication the ledger exists to prevent. IMAP UIDs are also only +> unique per-folder, not per-account. +> +> **3. The named seam is the wrong shape.** +> §5 and §7 place the IMAP adapter "behind the existing `InboundEmailProvider` +> seam." That interface is webhook-shaped in both methods — +> `verifySignature(request: Request)` and `receiveDelivery(request: Request)`. +> A cron-driven fetch has no `Request`. The *data* contract +> (`RawInboundMessage`, raw bytes, a single `parseInboundEmail`) is +> satisfiable; the *interface* is not. The claims "least new code" and "not new +> fetch code" both rest on this and are therefore overstated. +> +> Until these are answered, treat §4–§7 below as a sketch of intent, not a +> buildable specification. §2's spike results and §3's provider matrix are +> independently verified and stand on their own. ## 1. The problem @@ -16,7 +58,9 @@ There is also no UI for connecting a mailbox. The operator runs a raw `curl` aga ## 2. What the spike established (2026-07-20, verified live) -**Scheduled fetch already ships.** `runGmailWatchMaintenance` (`src/mail/gmail-watch-maintenance.ts`, the daily cron in `vercel.json`) performs "a bounded reconciliation sweep" that enqueues *the same reconcile job the push path enqueues*. That job (`src/mail/gmail-reconcile.ts`) reads the mailbox's **stored cursor** — explicitly never the push notification's `historyId` — then calls `history.list` followed by `messages.get?format=raw`. It consumes nothing from the push payload. +**Scheduled fetch already ships.** `runGmailWatchMaintenance` (`src/mail/gmail-watch-maintenance.ts`, the daily cron in `vercel.json`) performs "a bounded reconciliation sweep" that enqueues *the same reconcile job the push path enqueues*. That job (`src/mail/gmail-reconcile.ts`) reads the mailbox's **stored cursor** — explicitly never the push notification's `historyId` — then calls `history.list` followed by `messages.get?format=raw`. It takes only `mailboxId` from the job payload; the payload's `historyId` is logged and never acted on. + +Note the field is a latent hazard rather than a clean equivalence: the webhook writes the notification's *new* watermark into `historyId`, while a sweep writes the *stored cursor* — semantically opposite values in the same field. Harmless while nothing reads it, and a trap for anything that later does. Consequence: **push only makes the same job run sooner.** Making scheduled fetch the primary intake is a scheduling change, not new fetch code. The lease around `history.list` (`claimReconcileLease`) already prevents concurrent runs from double-fetching. diff --git a/specs/modules/catalog.md b/specs/modules/catalog.md index 27016e5..08650c3 100644 --- a/specs/modules/catalog.md +++ b/specs/modules/catalog.md @@ -73,7 +73,10 @@ passkeys and tags. ### 2.3 Not ported — obsolete by architecture -Move/Remove IMAP Message (no IMAP polling) · Faster Search via Meilisearch (search is +Move/Remove IMAP Message (no server-side folder manipulation; note the "no IMAP +polling" rationale this once carried is void as of HT-92 — scheduled fetch is now +permitted, so if IMAP intake ships this entry needs re-deciding on its own merits) · +Faster Search via Meilisearch (search is Postgres-native) · Auto Login from notification emails · Custom Homepage · Ticket Number in subject (threading never depends on it; display numbers shipped in HT-27) · Mailbox Icons · Twitter/X DM integration (API effectively dead). diff --git a/src/providers/inbound-email.ts b/src/providers/inbound-email.ts index 9982008..0b6bc35 100644 --- a/src/providers/inbound-email.ts +++ b/src/providers/inbound-email.ts @@ -3,11 +3,23 @@ * webhooks. * * See `src/providers/README.md` for the pattern this fits into. Per - * CHARTER.md §2/§4, inbound mail arrives via **push webhooks**, never IMAP - * polling — "no daemons, no polling loops" applies to inbound mail first - * and foremost. This is where Gmail-push-via-Pub/Sub plugs in today, and - * where later providers (Postmark inbound, SES inbound, a forwarding- - * address transport, ...) plug in without the engine changing. + * CHARTER.md §2/§4 (as amended 2026-07-20, HT-92), inbound mail arrives + * either by **push webhook** or by a **bounded, stateless scheduled fetch** — + * the constraint is "no daemons, no long-running processes," not "no + * fetching on a schedule." Nothing may stay resident either way: no IMAP + * IDLE, no held connections, no worker loops. + * + * **This interface is the WEBHOOK-shaped half of that seam** — both its + * methods take an HTTP `Request`, so a cron-driven fetch (which has no + * `Request`) does not fit here and must not be forced into it. Adding a + * scheduled-fetch transport means a sibling seam sharing this module's DATA + * contract (`RawInboundMessage`, raw bytes, one `parseInboundEmail`), not an + * implementation of this interface. That gap is called out in + * `specs/mail/mailbox-connection.md` §2 and is unresolved. + * + * This is where Gmail-push-via-Pub/Sub plugs in today, and where later + * webhook providers (Postmark inbound, SES inbound, ...) plug in without the + * engine changing. * * ## Raw bytes in, nothing pre-parsed * diff --git a/src/providers/queue.ts b/src/providers/queue.ts index f9649a7..da8bd12 100644 --- a/src/providers/queue.ts +++ b/src/providers/queue.ts @@ -8,8 +8,10 @@ * * Serverless queue consumers are **push-delivered**, not pulled: the * platform invokes an HTTP handler with a queued message, rather than the - * engine running a loop that polls for work (the charter's "no daemons, no - * polling loops" principle). This interface models that shape directly — + * engine running a resident loop that polls for work (the charter's "no + * daemons, no long-running processes" principle — a bounded invocation that + * reads and exits is permitted; a process that stays up is not). This + * interface models that shape directly — * there is no `dequeue`/`poll` method. The handler side of the contract is * `QueueMessageHandler`, invoked by adapter glue that receives the * platform's webhook/invocation and adapts it into a `QueueMessage`. From 55383ec8715a5019b60a29faaff96c3ec1475080 Mon Sep 17 00:00:00 2001 From: TJ Baker <1617679+zaridan@users.noreply.github.com> Date: Mon, 20 Jul 2026 14:13:51 -0700 Subject: [PATCH 3/5] docs(spec): close CodeRabbit findings on the mailbox-connection spec (HT-92) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Six of nine actionable findings taken; three were wording preferences, answered on the PR rather than applied. All land on the spec — the charter amendment drew none. Two were claims that were wrong, not merely thin: - "Outbound needs no DNS" overclaimed. Sending through the operator's own server inherits whatever SPF/DKIM/DMARC that domain already has; it does not guarantee deliverability. A domain with no SPF or a broken selector delivers just as badly through Helpthread as through its owner's normal client. The true claim is that we add no NEW DNS burden, unlike every provider-webhook alternative. - "Errors surface verbatim" was a leak. Raw IMAP/SMTP rejection text routinely carries hostnames, account addresses and auth detail; echoing it to a browser is not acceptable. Errors are now mapped to sanitized messages with diagnostic codes, raw text to redacted server logs only. Four were real gaps: - `Send test email` could manufacture a customer conversation. It now must address the connected mailbox itself (never an operator-typed recipient, so the setup screen cannot become a way to mail third parties), carry a correlation nonce the pipeline drops on, and be rate-limited. - No timeouts were specified. Every IMAP/SMTP operation now carries a timeout derived from the remaining invocation budget, the worker stops starting work it cannot finish, and connections close on success, timeout and failure alike. Without this one hung FETCH burns the whole invocation, every minute, indefinitely. - The UID cursor advances on COMMIT, not on fetch, and holds at the last committed UID on partial failure. Re-fetching a committed message is harmless because ingest is idempotent; advancing past an uncommitted one loses mail silently, which §2 does not permit. - App-password lifecycle was undocumented. They die quietly — no revocation signal, no refresh failure — when the account password changes, an admin disables them, or the account enrols in Advanced Protection. Auth failure now maps to the same `needs_reconnect` state a dead OAuth grant produces, and the reconnect path differs per provider (credential re-entry, not a consent redirect). Also noted: encrypting the credential at rest is not sufficient without deny-by-default server-only authorization — the same gap already open on mailbox_oauth_tokens, tracked in §8. The spec remains DRAFT; these are quality fixes to a document still blocked on its three mail-semantics questions. Co-Authored-By: Claude Opus 4.8 --- specs/mail/mailbox-connection.md | 49 +++++++++++++++++++++++++++++--- 1 file changed, 45 insertions(+), 4 deletions(-) diff --git a/specs/mail/mailbox-connection.md b/specs/mail/mailbox-connection.md index 18551f6..bbe5ed9 100644 --- a/specs/mail/mailbox-connection.md +++ b/specs/mail/mailbox-connection.md @@ -70,7 +70,9 @@ Consequence: **push only makes the same job run sooner.** Making scheduled fetch Worth recording: HT-49 exists because Gmail's `users.messages.send` **API** rewrites the `Message-ID` we set, which is why the reply token had to move into `References` (`src/mail/send.ts`). SMTP submission did not rewrite it. One observation is not proof of a general rule, and the `References` mechanism works on both transports — so this is a point in SMTP's favour, not a reason to change the threading model. -**Outbound needs no DNS.** Because replies go through the operator's own mail server, they are signed by that server's DKIM, from its IPs, on a domain whose SPF already authorises it. Every provider-webhook alternative considered (Postmark, Resend, SES, Cloudflare) would reintroduce SPF/DKIM/DMARC setup. This transport does not. +**Outbound needs no NEW DNS.** Replies go through the operator's own mail server, so they are signed by whatever DKIM that server already uses, from its IPs, under its existing SPF record. Every provider-webhook alternative considered (Postmark, Resend, SES, Cloudflare) requires the operator to add records for a *new* sending identity; this transport requires none. + +Stated precisely, because the earlier wording overclaimed: this does **not** guarantee deliverability. It inherits whatever the operator's domain already has. A domain with no SPF, a broken DKIM selector, or a `p=reject` DMARC record misaligned with its own sender will deliver just as badly through Helpthread as it does through the operator's normal mail client — the point is that we add no new DNS burden, not that we fix an existing one. Where deliverability is already broken, that is the operator's pre-existing mail configuration and should be diagnosed as such rather than as a Helpthread fault. ## 3. Provider support @@ -105,7 +107,25 @@ Behavioural contract: - **Method availability is provider-dependent.** For an M365 business domain the app-password option is disabled *with an explanation*, never silently absent. - **`Check connection`** performs a real IMAP login + `SELECT INBOX` and a real SMTP handshake + `AUTH`, reporting each leg independently. It must never report success from a config-shape check alone. - **`Send test email`** sends an actual message through the operator's SMTP and confirms it is observed back through the fetch path. This round trip is the single most valuable element on the screen, and it is what today's setup path has no equivalent of — which is why silent misconfiguration currently survives to production. -- **Errors surface verbatim** — the provider's actual IMAP/SMTP rejection text, plus a mapped hint for the common cases (2SV not enabled, app passwords blocked by admin policy, wrong port, mailbox not found). Never a bare "connection failed." + + It must not be able to manufacture a customer conversation. Requirements: + - **Addressed to the connected mailbox itself**, never to an operator-typed + recipient — a setup screen must not become a way to send mail to arbitrary + third parties. + - **Carries a unique correlation marker** (a nonce in a custom header and in + the minted `Message-ID`) that the screen polls for. + - **The ingestion pipeline recognises and drops it** on that marker, at the + same point it suppresses our own outbound echo. A test message must never + create a conversation, and must not be re-processed on later ticks. + - **Rate-limited**, so the button cannot be used as a send amplifier. +- **Errors are mapped, never echoed raw.** The user sees a sanitized, escaped + message plus a diagnostic code for the common cases (2SV not enabled, app + passwords blocked by admin policy, wrong port, mailbox not found, auth + rejected). The provider's raw IMAP/SMTP text goes to server-side logs only, + redacted — it routinely carries hostnames, account addresses, and + authentication detail that must not reach a browser. Never a bare + "connection failed" either: an unmapped failure shows its diagnostic code so + a support conversation can start from something specific. ## 5. Transport behaviour @@ -114,8 +134,9 @@ Behavioural contract: A cron entry alongside the existing four in `vercel.json`. Each invocation: connect, `SELECT INBOX`, fetch messages above the stored UID cursor, hand raw bytes to the pipeline, persist the new cursor, disconnect. - **No IDLE, no held connections, no resident process.** The connection opens and closes within the invocation. This is the charter constraint, and it must not be optimised away for latency later. -- **Track `UIDVALIDITY` alongside the UID cursor.** If the server changes `UIDVALIDITY`, every stored UID is meaningless and the cursor must be rebuilt. Skipping this silently drops or re-ingests mail. -- **Bound the batch.** `maxDuration` is 50 s (`vercel.json`). Cap messages per invocation and continue on the next tick rather than risk a timeout mid-fetch. +- **Track `UIDVALIDITY` alongside the UID cursor.** If the server changes `UIDVALIDITY`, every stored UID is meaningless and the cursor must be rebuilt. Skipping this silently drops or re-ingests mail. Rebuilding safely depends on the unresolved `providerMessageId` question at the top of this document — a UID-keyed ledger cannot survive the reset it is supposed to recover from. +- **The cursor advances on COMMIT, not on fetch.** A UID may only move the stored cursor once that message has been durably committed by the ingestion pipeline. On a partial failure the cursor stays at the last committed UID and the batch is retried — re-fetching an already-committed message is harmless (ingest is idempotent), whereas advancing past an uncommitted one loses mail silently, which the mail-semantics invariant does not permit. +- **Bound the batch, and bound the clock.** `maxDuration` is 50 s (`vercel.json`). Cap messages per invocation and continue on the next tick rather than risk a timeout mid-fetch. Separately, every network operation — IMAP connect, login, `SELECT`, each `FETCH`, and every SMTP step — carries its own timeout derived from the remaining invocation budget, and the worker stops *starting* new work once too little budget remains to finish it. Connections are closed on success, timeout, and failure alike. Without this a single hung `FETCH` consumes the whole invocation and the tick accomplishes nothing, every minute, indefinitely. - **Fetch raw.** `FETCH BODY.PEEK[]` — full RFC822, `.PEEK` so `\Seen` is not set. This satisfies `src/providers/inbound-email.ts` natively: raw bytes in, parsed exactly once by `parseInboundEmail`. No IMAP library's convenience parser may touch the message. ### Outbound — the operator's own SMTP @@ -126,6 +147,26 @@ Stated explicitly in the operator docs, because it is the quiet advantage: repli App passwords are long-lived secrets granting full mailbox access. They must be encrypted at rest via the existing `HELPTHREAD_TOKEN_ENC_KEY` path (`src/store/token-crypto.ts`, AES-256-GCM) already used for OAuth tokens, never logged, never returned by any API read, and write-only in the UI — show a "configured" state, never the value. +**Encryption is not sufficient on its own.** The credential table needs +deny-by-default server-only authorization, not merely ciphertext at rest — the +same gap already open on `mailbox_oauth_tokens` (see §8: RLS is disabled on +every table today). Whatever answer that gets must cover this table from the +day it exists, rather than inheriting the same debt. + +**Lifecycle — app passwords die quietly.** Unlike an OAuth grant, there is no +revocation signal and no refresh failure to classify. They stop working when +the account owner changes their password, when a Workspace admin disables app +passwords or enforces security-key-only 2SV, or when the account enrols in +Advanced Protection. The connection simply starts failing authentication. So: +- Auth failure on a scheduled fetch marks the mailbox `needs_reconnect`, the + same state a dead OAuth grant produces, so one operator-facing concept covers + both. +- The reconnect path is re-entering a credential, not a consent redirect — the + screen must say which, per provider, rather than offering an OAuth button to + a Fastmail user. +- This asymmetry is worth stating in the operator docs: OAuth is revocable and + observable; an app password is neither, and its failure mode is silence. + Note the asymmetry worth telling operators about: an app password cannot be scoped and does not expire; an OAuth token is scoped to `gmail.readonly` + `gmail.send` and is revocable from the account. ### Operational From dfc467f56dae717cab639671184d4ab532a08cdf Mon Sep 17 00:00:00 2001 From: TJ Baker <1617679+zaridan@users.noreply.github.com> Date: Mon, 20 Jul 2026 14:19:00 -0700 Subject: [PATCH 4/5] docs: close the last two CodeRabbit Major findings (HT-92) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit I triaged these two as wording preferences and skipped them. Both are labelled Major and both are real; one is an overclaim repeated throughout this PR and in the surrounding discussion. **"Scheduled fetch already ships" was misleading.** What ships is GMAIL history reconciliation — a cursor-based history.list walk against the Gmail API. That proves cursor-driven, cron-triggered intake is architecturally sound and running in production, which is what the amendment rests on. It provides nothing toward the IMAP UID-cursor fetch this spec describes: that adapter, its cursor semantics, its UIDVALIDITY handling and its idempotency key are all new and substantial work. The old phrasing invited reading the whole thing as a scheduling-only change. Now stated as "the pattern is proven," never "the transport is built." **"Drop the Pub/Sub dependency" read as removing support.** Both documents said drop while the charter says push stays preferred — a direct contradiction. Pub/Sub becomes optional, not unsupported: the push adapter, the webhook and the watch() renewal all remain fully supported for operators who want sub-minute latency. What changes is that they stop being mandatory setup. Corrected in the spec's build order and in the amendment's "what it unlocks" paragraph, and the build-order item now also notes it is Gmail-only and does nothing for IMAP. All nine CodeRabbit findings on this PR are now addressed. Every one landed on the spec; the amendment itself drew none. Co-Authored-By: Claude Opus 4.8 --- CHARTER.md | 2 +- specs/mail/mailbox-connection.md | 6 ++++-- 2 files changed, 5 insertions(+), 3 deletions(-) diff --git a/CHARTER.md b/CHARTER.md index 3ff895c..993a3e4 100644 --- a/CHARTER.md +++ b/CHARTER.md @@ -106,7 +106,7 @@ Solo-maintainer, BDFL model for now — there is one project, one person account Rationale: the original wording conflated two separable things — a *daemon* (a process that stays resident and holds an open connection, which is what makes FreeScout require a persistent PHP host) and a *scheduled fetch* (an invocation that ends). Only the first is incompatible with serverless. §4's Phase 1 language already permitted "bounded reconciliation fetches, never a long-running poller," and the daily watch-maintenance cron has always been exactly that — it enqueues the same reconcile job the push path enqueues, and that job reads the mailbox's *stored cursor*, never the push notification's `historyId`. The engine already did on a schedule what §2 appeared to forbid; this amendment removes an internal inconsistency rather than introducing a capability. -What it unlocks: dropping the Pub/Sub dependency (six setup steps, including the two that fail silently — the domain-restricted-sharing org-policy block and the missing `serviceAccountTokenCreator` grant), and IMAP/SMTP mailbox connection via app password, which needs no DNS records, no MX change, no cloud console, and no third-party provider account. Measured against the ~26-step, 4-console setup an adopter faces today (HT-82 dogfood), this is the difference between Helpthread being evaluable and not being evaluated. §1's own standard — *operators should find running the thing dead easy* — was not being met. +What it unlocks: making Pub/Sub **optional rather than mandatory** — six setup steps an operator no longer has to complete, including the two that fail silently (the domain-restricted-sharing org-policy block and the missing `serviceAccountTokenCreator` grant). Gmail push itself remains fully supported and preferred where an operator wants sub-minute latency; only its status as a setup prerequisite changes. It also unlocks IMAP/SMTP mailbox connection via app password, which needs no DNS records, no MX change, no cloud console, and no third-party provider account. Measured against the ~26-step, 4-console setup an adopter faces today (HT-82 dogfood), this is the difference between Helpthread being evaluable and not being evaluated. §1's own standard — *operators should find running the thing dead easy* — was not being met. Verified before amending (2026-07-20, live against the dogfood desk): IMAP + app password authenticated `help@resonantiq.app` and returned raw RFC822 with a real reply's `ht.ht1.…` signed token intact in its `References` chain, connection closed on exit; SMTP + the same credential sent with our own `Message-ID` preserved verbatim through Gmail's submission path. diff --git a/specs/mail/mailbox-connection.md b/specs/mail/mailbox-connection.md index bbe5ed9..47f4862 100644 --- a/specs/mail/mailbox-connection.md +++ b/specs/mail/mailbox-connection.md @@ -58,7 +58,9 @@ There is also no UI for connecting a mailbox. The operator runs a raw `curl` aga ## 2. What the spike established (2026-07-20, verified live) -**Scheduled fetch already ships.** `runGmailWatchMaintenance` (`src/mail/gmail-watch-maintenance.ts`, the daily cron in `vercel.json`) performs "a bounded reconciliation sweep" that enqueues *the same reconcile job the push path enqueues*. That job (`src/mail/gmail-reconcile.ts`) reads the mailbox's **stored cursor** — explicitly never the push notification's `historyId` — then calls `history.list` followed by `messages.get?format=raw`. It takes only `mailboxId` from the job payload; the payload's `historyId` is logged and never acted on. +**GMAIL scheduled reconciliation already ships — not a generic scheduled fetch.** This distinction is load-bearing and the earlier wording blurred it. What exists today is Gmail *history* reconciliation: a cursor-based `history.list` walk against the Gmail API. It proves that cursor-driven, cron-triggered intake is architecturally sound and already in production, which is what the charter amendment rests on. It does **not** provide a byte of the IMAP UID-cursor fetch this document specifies — that adapter, its cursor semantics, its `UIDVALIDITY` handling, and its idempotency key are all new work, and substantial. Read the paragraph below as "the pattern is proven," never as "the transport is built." + +`runGmailWatchMaintenance` (`src/mail/gmail-watch-maintenance.ts`, the daily cron in `vercel.json`) performs "a bounded reconciliation sweep" that enqueues *the same reconcile job the push path enqueues*. That job (`src/mail/gmail-reconcile.ts`) reads the mailbox's **stored cursor** — explicitly never the push notification's `historyId` — then calls `history.list` followed by `messages.get?format=raw`. It takes only `mailboxId` from the job payload; the payload's `historyId` is logged and never acted on. Note the field is a latent hazard rather than a clean equivalence: the webhook writes the notification's *new* watermark into `historyId`, while a sweep writes the *stored cursor* — semantically opposite values in the same field. Harmless while nothing reads it, and a trap for anything that later does. @@ -183,7 +185,7 @@ That is a configuration detail on the mailbox record, not a transport concern. ## 7. Build order -1. **Drop the Pub/Sub dependency** — reschedule the existing reconcile sweep. Least new code, largest reduction in setup steps, removes both silent-failure traps and the billing requirement (Pub/Sub is what forced it). +1. **Make Pub/Sub optional — not unsupported.** Reschedule the existing Gmail reconcile sweep so it is the primary transport, and let the engine boot with no `GMAIL_PUBSUB_*` vars. The Gmail push adapter, the webhook, and the `watch()` renewal all remain fully supported for operators who want sub-minute latency; what changes is that they stop being *mandatory setup*. This removes both silent-failure traps and the billing requirement (Pub/Sub is what forced it). Least new code of any item here — and note this is Gmail-only, per §2: it does nothing for IMAP. 2. **IMAP provider adapter** behind the existing `InboundEmailProvider` seam, with fixtures proving equivalence against the Gmail path. 3. **Migrations** — UID cursor + `UIDVALIDITY`; encrypted mailbox credential. 4. **SMTP sender** behind the existing `EmailSender` seam. From 7e7618206380d0ee6ca2f9e2f8f1176f55b25fe4 Mon Sep 17 00:00:00 2001 From: TJ Baker <1617679+zaridan@users.noreply.github.com> Date: Mon, 20 Jul 2026 16:07:15 -0700 Subject: [PATCH 5/5] =?UTF-8?q?docs(charter):=20neither=20transport=20is?= =?UTF-8?q?=20primary=20=E2=80=94=20the=20operator=20chooses=20(HT-92/HT-1?= =?UTF-8?q?00)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The amendment as drafted elevated scheduled fetch to a 'first-class primary transport, not merely a reconciliation backstop.' That framing was the drafting assistant's, not TJ's — it was explicitly named as his call and then made for him ('That's a business call, not an engineering one, and it's yours'). On review TJ pointed out the framing is wrong at the root: there is no system-wide default, because the operator picks the transport at setup. Verified against specs/deploy/gmail-inbound-runbook.md Part A — Gmail push needs a GCP project, an Internal OAuth app, a Pub/Sub topic, a Gmail publisher grant, a service account, and an OIDC push subscription, plus two documented org-policy landmines hit during live provisioning. Scheduled fetch needs an app password or OAuth. The easier path wins on adoption regardless of what any document calls primary. So neither is designated primary. Both are fully supported intake paths and the charter now states the tradeoff plainly — latency against setup cost — and leaves the choice with the operator. TJ's approved §1 positioning line is unchanged and never claimed a primary: 'push-based delivery where providers offer it, bounded scheduled fetches where they don't — and no resident process either way.' The §7 appendix amendment now records what he actually ratified, with his own words, and notes that the 'first-class primary transport' framing was removed as assistant-authored. Two stale 'primary' claims in the mailbox-connection draft reconciled to match. Co-Authored-By: Claude Opus 4.8 --- CHARTER.md | 6 ++++-- specs/mail/mailbox-connection.md | 4 ++-- 2 files changed, 6 insertions(+), 4 deletions(-) diff --git a/CHARTER.md b/CHARTER.md index 993a3e4..98cc24a 100644 --- a/CHARTER.md +++ b/CHARTER.md @@ -15,7 +15,7 @@ Helpthread is the answer for people who already live on serverless infrastructur - **Innovate on the platform. Be boringly faithful on mail semantics.** Everything about *how* Helpthread runs — compute model, storage, deployment — is fair game for rethinking from scratch. How it parses, threads, and sends email is not. Mature systems — FreeScout and Chatwoot among them — have earned a decade of edge-case scars in production email handling; we respect that earned knowledge, reproduce proven behavior, and verify against fixtures rather than re-deriving from first principles. Any change to mail behavior needs proof of equivalence or an explicit, written justification — not a hunch that the old way looked wrong. During this project's own early development we watched well-intentioned "improvements" to mail handling silently destroy message content in testing; that lesson is why this rule exists and why it's not up for debate per-PR. - **Threading authority lives on the outbound side.** Inbound threading headers (`References`, `In-Reply-To`) are written by every mail client on earth, inconsistently — they cannot be trusted. The one header the engine fully controls is the Message-ID it emits. So threading is anchored there: signed reply tokens in outbound Message-IDs, authenticating replies on their way back in — a pattern as old as mailing-list software. This is the mechanism the whole system leans on; treat it accordingly. - **API-first — and the UI isn't the only client.** Anything the UI can do, a typed public API can do. The UI is a client of that API, not a special case; so are AI agents. An MCP server exposing the same operations ships as a first-class client of the same API, so any operator can point their own AI tooling at their own helpdesk. -- **Serverless-native, Vercel-first, not Vercel-only.** No daemons, no long-running processes. Inbound mail arrives either by push webhook (Gmail push through Pub/Sub, and equivalents as we add providers) or by a bounded, stateless scheduled fetch — a function that connects, reads from a stored cursor, and exits. Push is preferred wherever a provider offers it, on latency grounds; scheduled fetch is a first-class primary transport, not merely a reconciliation backstop. Nothing stays resident either way: no held connections, no IMAP IDLE, no worker loops. Anything that needs to happen later is a scheduled action or a cron trigger, not a process sitting in a loop. The engine's core reaches every platform service through interfaces the project owns — see the platform posture note in the architecture section. *(Amended 2026-07-20 — scheduled-fetch intake; see §7 appendix.)* +- **Serverless-native, Vercel-first, not Vercel-only.** No daemons, no long-running processes. Inbound mail arrives either by push webhook (Gmail push through Pub/Sub, and equivalents as we add providers) or by a bounded, stateless scheduled fetch — a function that connects, reads from a stored cursor, and exits. **Neither is "the primary" — the operator chooses at setup, trading latency against setup cost:** push is near-instant but needs a GCP project and roughly six provisioning steps (`specs/deploy/gmail-inbound-runbook.md` Part A); scheduled fetch costs up to a cron interval of latency and needs an app password or OAuth. Both are fully supported intake paths; neither is a fallback for the other. Nothing stays resident either way: no held connections, no IMAP IDLE, no worker loops. Anything that needs to happen later is a scheduled action or a cron trigger, not a process sitting in a loop. The engine's core reaches every platform service through interfaces the project owns — see the platform posture note in the architecture section. *(Amended 2026-07-20 — scheduled-fetch intake; see §7 appendix.)* - **Own your data.** Self-hosted on the operator's own Vercel and Supabase accounts. Conversation data never touches Resonant IQ-operated infrastructure — we don't sit in the data path — and AI features preserve that: assistants call model providers the operator configures, with the operator's own keys. **The promise is absolute and carries no exception.** *(The 2026-07-19 "managed hosting" carve-out was **rescinded 2026-07-20** on TJ's review — he had selected managed hosting from a picker that labelled it "Recommended", and rejected the model once its consequences were clear. See the §7 appendix.)* ## 3. Licensing & intellectual property @@ -102,7 +102,9 @@ Solo-maintainer, BDFL model for now — there is one project, one person account **Amended 2026-07-19 (HT-75):** §3 and §4 — the knowledge base reclassified from free-forever core to a paid first-party module. FreeScout's own reference instance runs its Knowledge Base as a paid purchase (`specs/ui/admin-ia.md`); keeping ours core was chosen 2026-07-18 as a marketed free-vs-FreeScout differentiator (`specs/modules/catalog.md` §1), and TJ reversed that call on review. Not a retroactive paywall under the §3 invariant — the knowledge base was never shipped (`specs/modules/catalog.md` §2.1's shipped list never included it). Full paid-module catalog: `specs/modules/catalog.md`. -**Amended 2026-07-20 (HT-92, scheduled-fetch intake):** §2 — the "Serverless-native" bullet's prohibition is restated from *"no daemons, no polling loops"* to **"no daemons, no long-running processes."** Inbound mail may arrive either by push webhook or by a **bounded, stateless scheduled fetch** — a function that connects, fetches from a stored cursor, and exits. Push remains preferred wherever a provider offers it, on latency grounds, but scheduled fetch is now a first-class primary transport rather than only a reconciliation backstop. Consequential wording updated at §1's opening summary, the FreeScout comparison, the serverless positioning line, and §4's Phase 1 description. +**Amended 2026-07-20 (HT-92, scheduled-fetch intake):** §2 — the "Serverless-native" bullet's prohibition is restated from *"no daemons, no polling loops"* to **"no daemons, no long-running processes."** Inbound mail may arrive either by push webhook or by a **bounded, stateless scheduled fetch** — a function that connects, fetches from a stored cursor, and exits. **Neither transport is designated primary.** The operator picks one at setup and trades latency against setup cost — push is near-instant but requires a GCP project and ~6 provisioning steps plus two known org-policy landmines (`specs/deploy/gmail-inbound-runbook.md` Part A); scheduled fetch costs up to a cron interval and requires an app password or OAuth. Consequential wording updated at §1's opening summary, the FreeScout comparison, the serverless positioning line, and §4's Phase 1 description. + +*Wording ratified by TJ, 2026-07-20.* He initiated the change — *"Wait, wait, wait - why would we forbit imap polling?"* and *"Wasn't the no polling clause just to say... we as Helpthread will never poll their code?"* — gated it on evidence (*"let's get this fully tested out before we change the charter. I want to see it in action first"*), and approved the §1 positioning line verbatim: *"push-based delivery where providers offer it, bounded scheduled fetches where they don't — and no resident process either way."* An earlier draft of this amendment elevated scheduled fetch to a *"first-class primary transport"*; that framing was the drafting assistant's, not TJ's, and was removed on review — the operator, not this charter, chooses the transport. Rationale: the original wording conflated two separable things — a *daemon* (a process that stays resident and holds an open connection, which is what makes FreeScout require a persistent PHP host) and a *scheduled fetch* (an invocation that ends). Only the first is incompatible with serverless. §4's Phase 1 language already permitted "bounded reconciliation fetches, never a long-running poller," and the daily watch-maintenance cron has always been exactly that — it enqueues the same reconcile job the push path enqueues, and that job reads the mailbox's *stored cursor*, never the push notification's `historyId`. The engine already did on a schedule what §2 appeared to forbid; this amendment removes an internal inconsistency rather than introducing a capability. diff --git a/specs/mail/mailbox-connection.md b/specs/mail/mailbox-connection.md index 47f4862..14deb17 100644 --- a/specs/mail/mailbox-connection.md +++ b/specs/mail/mailbox-connection.md @@ -64,7 +64,7 @@ There is also no UI for connecting a mailbox. The operator runs a raw `curl` aga Note the field is a latent hazard rather than a clean equivalence: the webhook writes the notification's *new* watermark into `historyId`, while a sweep writes the *stored cursor* — semantically opposite values in the same field. Harmless while nothing reads it, and a trap for anything that later does. -Consequence: **push only makes the same job run sooner.** Making scheduled fetch the primary intake is a scheduling change, not new fetch code. The lease around `history.list` (`claimReconcileLease`) already prevents concurrent runs from double-fetching. +Consequence: **push only makes the same job run sooner.** Making scheduled fetch a fully-supported intake path (rather than only a reconciliation backstop) is a scheduling change, not new fetch code. The lease around `history.list` (`claimReconcileLease`) already prevents concurrent runs from double-fetching. **IMAP + app password works.** Probed `help@resonantiq.app` directly: authenticated, `SELECT INBOX` (`UIDVALIDITY=1`, `UIDNEXT=32`, `EXISTS=29`), `UID FETCH BODY.PEEK[]` returned raw RFC822 for five messages in 1174 ms, connection closed on exit. A real reply's `References` chain carried the `ht.ht1.…` signed reply token intact — the CHARTER.md §2 threading anchor holds on this transport. @@ -185,7 +185,7 @@ That is a configuration detail on the mailbox record, not a transport concern. ## 7. Build order -1. **Make Pub/Sub optional — not unsupported.** Reschedule the existing Gmail reconcile sweep so it is the primary transport, and let the engine boot with no `GMAIL_PUBSUB_*` vars. The Gmail push adapter, the webhook, and the `watch()` renewal all remain fully supported for operators who want sub-minute latency; what changes is that they stop being *mandatory setup*. This removes both silent-failure traps and the billing requirement (Pub/Sub is what forced it). Least new code of any item here — and note this is Gmail-only, per §2: it does nothing for IMAP. +1. **Make Pub/Sub optional — not unsupported.** Reschedule the existing Gmail reconcile sweep so it can carry intake on its own, and let the engine boot with no `GMAIL_PUBSUB_*` vars. Neither transport is designated primary — per CHARTER §2, the operator chooses at setup. The Gmail push adapter, the webhook, and the `watch()` renewal all remain fully supported for operators who want sub-minute latency; what changes is that they stop being *mandatory setup*. This removes both silent-failure traps and the billing requirement (Pub/Sub is what forced it). Least new code of any item here — and note this is Gmail-only, per §2: it does nothing for IMAP. 2. **IMAP provider adapter** behind the existing `InboundEmailProvider` seam, with fixtures proving equivalence against the Gmail path. 3. **Migrations** — UID cursor + `UIDVALIDITY`; encrypted mailbox credential. 4. **SMTP sender** behind the existing `EmailSender` seam.