Skip to content

feat: move integration freshness sync onto Inngest - #20

Merged
scibly merged 3 commits into
knowledge-syncfrom
claude/ponytail-ultra-947015
Aug 28, 2026
Merged

feat: move integration freshness sync onto Inngest#20
scibly merged 3 commits into
knowledge-syncfrom
claude/ponytail-ultra-947015

Conversation

@NiclasDev63

@NiclasDev63 NiclasDev63 commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

Closes #10.

The scheduled freshness poll ran as a Vercel cron that took a singleton lease row and then chained POSTs to itself, hop by hop, until every due connection had had its turn. The lease was there because two overlapping chains would double the provider quota and race the same watermark; the hops were there because one serverless invocation could not outlive the work.

Inngest already solves both, so the mechanism goes and the domain rules stay.

Shape

  • integration-sync — cron 0 4 * * *. Lists due connections, sends one scibly/integration-poll.requested per connection. Nothing else.
  • integration-poll — one run per connection. Refreshes, polls the provider for pages modified since the watermark, marks matching sources stale, advances the watermark. Throws on failure, because the throw is what Inngest retries.
    • retries: 2, so three attempts.
    • concurrency: { key: "event.data.provider", limit: 3 } — the quota ceiling, previously implicit in the chain being serial.
    • onFailure records the failure and applies the backoff once, after the retries are spent, not once per attempt.

The event carries { connectionId, provider } and no credential.

Deleted

IntegrationSyncLease (+ migration), /api/cron/sync-integrations and its route guard, the crons entry in vercel.json, and CRON_SECRET — nothing reads it once the last cron route is gone. Net −800 lines.

Behaviour

Unchanged, deliberately: same 04:00 schedule, same least-recently-attempted-first selection, same lapsed-subscription exclusion, same backoff tiers, same watermark semantics (the watermark takes the instant the poll started, so an edit made mid-poll is caught by the next one rather than missed).

Verified against a real Notion workspace

Beyond the unit tests, run end to end on a live dev stack:

Fan-out one event per due connection; backed-off and lapsed connections got none
Real poll Notion API called, the connection's source marked stale, watermark advanced, consecutiveFailures cleared
Watermark second run's window narrowed to lastPolledAt − 1min, returned nothing, left staleAt alone
Non-matching source same poll, not in the result set, correctly not marked
Failure 3 attempts → one onFailure run → consecutiveFailures +1 (not +3), nextPollAfter at the exact tier, lastPolledAt untouched
Empty connection wrote lastAttemptedAt only, never contacted the provider

Notes for review

  • Functions are feature-owned and collected in apps/app/src/server/inngest.ts, the composition root for background work, the way api/root.ts collects tRPC routers. The architecture/boundaries rule forbids src/lib/ importing src/features/, so the collection cannot sit beside the Inngest client.
  • Out of ticket scope: removing CRON_SECRET. It was only ever read by the route this PR deletes. Say the word and I'll put it back.
  • Third commit removes the token-refresh scaffolding the ticket's refresh step would have used. refreshTokenEncrypted and tokenExpiresAt were write-only — set in the OAuth callback, read nowhere — and always NULL, since Notion's exchangeCode returns neither and Notion is the only provider in the registry. BaseIntegrationProvider.refreshToken was a stub whose whole behaviour was to throw, with no caller. Notion tokens don't expire; they die on revocation, which no refresh fixes, and the retry-then-back-off path already handles that correctly. Whoever adds a provider that needs a refresh adds these back with the code that reads them.
  • The second commit is an unrelated schema doc-comment cleanup, kept separate for a readable diff.

🤖 Generated with Claude Code

NiclasDev63 and others added 2 commits August 28, 2026 22:56
The scheduled poll ran as a Vercel cron that grabbed a singleton lease row
and then chained POSTs to itself, hop by hop, until every due connection had
had its turn. The lease existed because two overlapping chains would double
the provider quota and race the same watermark; the hops existed because one
serverless invocation could not outlive the work.

Inngest already solves both. A cron function lists the due connections and
sends one event each; a per-connection function does the actual poll. Retries,
the concurrency cap that keeps a provider from being hammered, and the
failure hook are declarations on the function rather than logic in the route.

Observable behaviour is unchanged: same 04:00 schedule, same selection order,
same backoff tiers, same watermark semantics.

Deleted with the mechanism they served: `IntegrationSyncLease`, the
`/api/cron/sync-integrations` route and its guard, the `crons` entry in
`vercel.json`, and `CRON_SECRET` — nothing reads it once the last cron route
is gone.

Functions are feature-owned and collected in `src/server/inngest.ts`, the
composition root for background work, mirroring how `api/root.ts` collects
tRPC routers. `src/lib/` may not import from `src/features/`, so the
collection cannot live beside the client.

Closes #10

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@vercel

vercel Bot commented Aug 28, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

2 Skipped Deployments
Project Deployment Actions Updated (UTC)
startup-app Ignored Ignored Aug 28, 2026 9:07pm
startup-web Ignored Ignored Aug 28, 2026 9:07pm

Request Review

`refreshTokenEncrypted` and `tokenExpiresAt` were write-only: set once in the
OAuth callback and read by nothing. Both were always NULL besides — Notion's
`exchangeCode` returns neither, and Notion is the only provider in the
registry. `BaseIntegrationProvider.refreshToken` was a stub whose entire
behaviour was to throw, with no caller.

Notion access tokens do not expire; they die when a user revokes access, and
no refresh call fixes that. A failing poll already retries and then backs off,
which is the right outcome for a connection that needs reconnecting by hand.

A provider that genuinely needs a refresh path can add these back together
with the code that reads them.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@scibly
scibly merged commit 2692749 into knowledge-sync Aug 28, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants