Keyed singleton scheduled tasks (scheduleTask) + migrate connectors off leak-prone renewal pattern#205
Merged
Merged
Conversation
…asks
Recurring/self-renewing jobs (watch & webhook renewals, polling, deferred
cleanup) were hand-managed with runTask({ runAt }) + a stored token +
cancelTask() before re-scheduling. That pattern is easy to get wrong: a
renewal callback that re-schedules itself, plus any extra entry into the
setup path (an onChannelEnabled re-dispatch, a re-init, a race), leaks
parallel self-perpetuating task chains that accumulate without bound and
eventually trip the runtime execution quota.
Add a keyed primitive that makes the leak impossible by construction:
- this.scheduleTask(key, callback, { runAt }) — schedules a singleton task
per key; re-scheduling under the same key atomically cancels-and-replaces
any pending task. At most one live scheduled task per key.
- this.cancelScheduledTask(key) — tears it down (no-op if none/already ran).
Exposed on the Tasks tool interface and as helpers on Tool/Twist. Docs
(RUNTIME.md periodic-work guidance, TOOLS_GUIDE.md scheduling section)
updated to steer recurring tasks to scheduleTask.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ed chains)
Replace the leak-prone runTask({ runAt }) + stored-token renewal/poll
pattern with the keyed singleton scheduleTask/cancelScheduledTask primitive
across every connector that runs a self-renewing or recurring scheduled task:
- Watch/subscription renewals: google-drive, google-calendar, google-chat,
ms-teams, jira, outlook-calendar. (google-drive & google-calendar also now
stop the prior provider watch before re-creating it, so a redundant setup
can't orphan a still-firing watch.)
- Periodic polls: airtable (poll + webhook renewal), google-tasks, asana,
apple-calendar — several of which previously had no token/cancel at all and
could fork unboundedly.
- Daily digests: slack (members + custom-emoji), and the per-channel daily
passes in ms-teams/google-chat.
Each chain is keyed per resource (e.g. watch-renewal:<id>, poll:<id>) so
re-scheduling replaces rather than forks, and disable paths cancel by key.
gmail and outlook-mail already cancel-before-reschedule (no leak) and keep
their existing self-heal logic. Adds a google-drive regression test.
Fixes the Google Drive connector auto-suspension (500+ worker invocations/24h
from accumulated renewal chains).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
A connector's self-renewing scheduled task (watch/webhook renewal, polling, daily digest) was hand-managed with
runTask({ runAt })+ a stored token +cancelTask()before re-scheduling. That pattern is easy to get wrong: a renewal callback that re-schedules itself, plus any extra entry into the setup path (anonChannelEnabledre-dispatch, a re-init, a race), leaks parallel self-perpetuating task chains that accumulate without bound and eventually trip the runtime execution quota.This is exactly what auto-suspended the Google Drive connector (≈350 → 3351 worker invocations/day, near-zero DB writes — a steady 24/7 background climb from accumulated renewal chains). google-calendar had the identical latent bug, and several pollers (airtable, google-tasks) had no token/cancel at all, so they could fork unboundedly with no way to stop them.
The leaky pattern was even documented as the recommended approach in
connectors/AGENTS.md, so connectors copied it faithfully.What
New SDK primitive (
@plotday/twister) — makes the leak impossible by construction:this.scheduleTask(key, callback, { runAt })— schedules a singleton task perkey; re-scheduling under the same key atomically cancels-and-replaces any pending task. At most one live scheduled task per key.this.cancelScheduledTask(key)— tears it down (no-op if none / already ran).Exposed on the Tasks tool interface and as
Tool/Twisthelpers. Changeset included (minor). The runtime side (thetask_keycolumn + atomic replace-on-create inCallbacksState, and the Tasks-tool wiring) ships in the matching core PR, which also bumps the submodule pointer to this branch.Docs — fixed the
connectors/AGENTS.mdwatch-renewal snippet (it was teaching the leaky pattern), added a checklist item + pitfall, and updatedtwister/docs/RUNTIME.md(periodic-work example) andTOOLS_GUIDE.md(scheduling section) to steer recurring tasks toscheduleTask.Connector migration — moved every connector with a self-renewing/recurring scheduled task onto the keyed primitive:
Each chain is keyed per resource (
watch-renewal:<id>,poll:<id>,members-sync:<channelId>, …) so re-scheduling replaces rather than forks, and disable paths cancel by key. Pollers that previously had no teardown now cancel on disable. google-drive/google-calendar additionally stop the prior provider watch before re-creating it (so a redundant setup can't orphan a still-firing watch). Added a google-drive regression test.Left as-is (not affected): gmail and outlook-mail already cancel-before-reschedule (no leak) and keep their self-heal logic; github's
runTaskoverride is a pass-through; linkedin's retry terminates.Backwards compatibility
scheduleTask/taskKeyare purely additive. Any pre-deploy in-flight (un-keyed) renewal task fires once more after deploy, then the chain becomes keyed — no lingering leak. Suspended connectors un-suspend on their next deploy.Verification
*_renewal_task/poll_task/cancelTaskreferences.🤖 Generated with Claude Code