Skip to content

Keyed singleton scheduled tasks (scheduleTask) + migrate connectors off leak-prone renewal pattern#205

Merged
KrisBraun merged 2 commits into
mainfrom
keyed-scheduled-tasks
Jun 17, 2026
Merged

Keyed singleton scheduled tasks (scheduleTask) + migrate connectors off leak-prone renewal pattern#205
KrisBraun merged 2 commits into
mainfrom
keyed-scheduled-tasks

Conversation

@KrisBraun

Copy link
Copy Markdown
Contributor

Why

A connector's self-renewing scheduled task (watch/webhook renewal, polling, daily digest) was hand-managed with runTask({ runAt }) + a stored token + cancelTask() before re-scheduling. That pattern is easy to get wrong: a renewal callback that re-schedules itself, plus any extra entry into the setup path (an onChannelEnabled re-dispatch, a re-init, a race), leaks parallel self-perpetuating task chains that accumulate without bound and eventually trip the runtime execution quota.

This is exactly what auto-suspended the Google Drive connector (≈350 → 3351 worker invocations/day, near-zero DB writes — a steady 24/7 background climb from accumulated renewal chains). google-calendar had the identical latent bug, and several pollers (airtable, google-tasks) had no token/cancel at all, so they could fork unboundedly with no way to stop them.

The leaky pattern was even documented as the recommended approach in connectors/AGENTS.md, so connectors copied it faithfully.

What

New SDK primitive (@plotday/twister) — makes the leak impossible by construction:

  • this.scheduleTask(key, callback, { runAt }) — schedules a singleton task per key; re-scheduling under the same key atomically cancels-and-replaces any pending task. At most one live scheduled task per key.
  • this.cancelScheduledTask(key) — tears it down (no-op if none / already ran).

Exposed on the Tasks tool interface and as Tool/Twist helpers. Changeset included (minor). The runtime side (the task_key column + atomic replace-on-create in CallbacksState, and the Tasks-tool wiring) ships in the matching core PR, which also bumps the submodule pointer to this branch.

Docs — fixed the connectors/AGENTS.md watch-renewal snippet (it was teaching the leaky pattern), added a checklist item + pitfall, and updated twister/docs/RUNTIME.md (periodic-work example) and TOOLS_GUIDE.md (scheduling section) to steer recurring tasks to scheduleTask.

Connector migration — moved every connector with a self-renewing/recurring scheduled task onto the keyed primitive:

Class Connectors
Watch/subscription renewal google-drive, google-calendar, google-chat, ms-teams, jira, outlook-calendar
Periodic poll airtable (+ webhook renewal), google-tasks, asana, apple-calendar
Daily digest slack (members + custom-emoji), ms-teams, google-chat

Each chain is keyed per resource (watch-renewal:<id>, poll:<id>, members-sync:<channelId>, …) so re-scheduling replaces rather than forks, and disable paths cancel by key. Pollers that previously had no teardown now cancel on disable. google-drive/google-calendar additionally stop the prior provider watch before re-creating it (so a redundant setup can't orphan a still-firing watch). Added a google-drive regression test.

Left as-is (not affected): gmail and outlook-mail already cancel-before-reschedule (no leak) and keep their self-heal logic; github's runTask override is a pass-through; linkedin's retry terminates.

Backwards compatibility

scheduleTask/taskKey are purely additive. Any pre-deploy in-flight (un-keyed) renewal task fires once more after deploy, then the chain becomes keyed — no lingering leak. Suspended connectors un-suspend on their next deploy.

Verification

  • All 13 touched connectors build + lint clean; no leftover *_renewal_task/poll_task/cancelTask references.
  • twister builds; changeset validates.
  • Runtime behavior (atomic replace / independent keys / cancel) is covered by an integration test in the core PR (5 tests), plus a google-drive connector regression test here.

🤖 Generated with Claude Code

KrisBraun and others added 2 commits June 17, 2026 00:44
…asks

Recurring/self-renewing jobs (watch & webhook renewals, polling, deferred
cleanup) were hand-managed with runTask({ runAt }) + a stored token +
cancelTask() before re-scheduling. That pattern is easy to get wrong: a
renewal callback that re-schedules itself, plus any extra entry into the
setup path (an onChannelEnabled re-dispatch, a re-init, a race), leaks
parallel self-perpetuating task chains that accumulate without bound and
eventually trip the runtime execution quota.

Add a keyed primitive that makes the leak impossible by construction:

- this.scheduleTask(key, callback, { runAt }) — schedules a singleton task
  per key; re-scheduling under the same key atomically cancels-and-replaces
  any pending task. At most one live scheduled task per key.
- this.cancelScheduledTask(key) — tears it down (no-op if none/already ran).

Exposed on the Tasks tool interface and as helpers on Tool/Twist. Docs
(RUNTIME.md periodic-work guidance, TOOLS_GUIDE.md scheduling section)
updated to steer recurring tasks to scheduleTask.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ed chains)

Replace the leak-prone runTask({ runAt }) + stored-token renewal/poll
pattern with the keyed singleton scheduleTask/cancelScheduledTask primitive
across every connector that runs a self-renewing or recurring scheduled task:

- Watch/subscription renewals: google-drive, google-calendar, google-chat,
  ms-teams, jira, outlook-calendar. (google-drive & google-calendar also now
  stop the prior provider watch before re-creating it, so a redundant setup
  can't orphan a still-firing watch.)
- Periodic polls: airtable (poll + webhook renewal), google-tasks, asana,
  apple-calendar — several of which previously had no token/cancel at all and
  could fork unboundedly.
- Daily digests: slack (members + custom-emoji), and the per-channel daily
  passes in ms-teams/google-chat.

Each chain is keyed per resource (e.g. watch-renewal:<id>, poll:<id>) so
re-scheduling replaces rather than forks, and disable paths cancel by key.
gmail and outlook-mail already cancel-before-reschedule (no leak) and keep
their existing self-heal logic. Adds a google-drive regression test.

Fixes the Google Drive connector auto-suspension (500+ worker invocations/24h
from accumulated renewal chains).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@KrisBraun KrisBraun merged commit 59cb509 into main Jun 17, 2026
1 check passed
@KrisBraun KrisBraun deleted the keyed-scheduled-tasks branch June 17, 2026 13:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant