Skip to content

Bug: Telegram polling TELEGRAM_POLL_RESTART_POLICY too aggressive — maxMs=30s causes restart storms #88256

Closed

Description

@survivor998

Summary

The TELEGRAM_POLL_RESTART_POLICY in dist/monitor-polling.runtime-*.js uses maxMs: 3e4 (30 seconds), which is too aggressive for network instability scenarios. During connectivity fluctuations, this causes rapid restart cycles that compound the problem rather than allowing graceful recovery.

Additionally:

  • Stop timeout has no circuit breaker — consecutive stop timeouts trigger immediate retries without cooldown
  • stopRunner/stopBot can be called repeatedly without deduplication, causing race conditions
  • Watchdog-triggered stops bypass the unified stop entry point

Current Behavior (v2026.5.28)

const TELEGRAM_POLL_RESTART_POLICY = {
  initialMs: 2e3,  // 2s — too fast after a crash
  maxMs: 3e4,       // 30s — too short for network recovery
  factor: 1.8,
  jitter: .25
};

Suggested Fix

  1. Harden restart policy: Increase initialMs to 3e4 and maxMs to 6e5 (10 minutes), with factor: 2 for proper exponential backoff:
const TELEGRAM_POLL_RESTART_POLICY = {
  initialMs: 3e4,  // 30s — give network time to stabilize
  maxMs: 6e5,       // 10 min — max wait before next attempt
  factor: 2,
  jitter: .2
};
  1. Add stop-timeout circuit breaker: Track consecutive stop timeouts with burst counting; when threshold exceeded, apply exponential cooldown before next restart cycle.

  2. Add requestStop() deduplication: Ensure stopRunner() and stopBot() are called at most once per polling cycle.

  3. Add noteHealthyPollingCycle() tracking: Reset circuit breaker state after successful polling cycles.

Environment

References

Activity

  1. clawsweeper commented on May 30, 2026

    @clawsweeper
    Contributor

    Thanks for the context here. I did a careful shell check against current main, and this is already implemented.

    Current main and shipped releases now implement the requested Telegram polling hardening: the restart policy is 30s/10m, repeated stop timeouts get a separate cooldown, stop requests are deduplicated, and healthy polling resets the backoff state. The June 9 follow-up overlaps with broader in-flight reply loss, but the specific polling restart-storm policy is already fixed by the merged Telegram polling PR.

    I found the merged PR that appears to have closed this: #88832: fix(telegram): slow polling restart storms.

    So I’m closing this as already implemented rather than keeping a duplicate issue open.

    Review details

    Best possible solution:

    Keep the shipped Telegram polling restart hardening, and leave any broader in-flight reply-loss investigation to #78380 or a narrower delivery issue.

    Do we have a high-confidence way to reproduce the issue?

    No for current main: the old 2s/30s policy was source-reproducible in the pre-fix code and user logs, but current main and shipped release source now contain the requested 30s/10m policy and stop-timeout cooldown. I did not run a live Telegram network-flap reproduction in this read-only close pass.

    Is this the best way to solve the issue?

    Yes: the merged fix is in the TelegramPollingSession owner boundary, covers normal and isolated polling paths, and has focused tests for the policy, cooldown, dedupe, and reset behavior.

    Security review:

    Security review: Non-PR issue cleanup; no patch security or supply-chain review applies.

    AGENTS.md: found and applied where relevant.

    What I checked:

    • Current source implements the requested restart policy and cooldown: Current main defines TELEGRAM_POLL_RESTART_POLICY as initialMs 30_000, maxMs 600_000, factor 2, jitter 0.2, and adds TELEGRAM_POLL_STOP_TIMEOUT_COOLDOWN_POLICY plus burst state in resolveTelegramRestartDelayMs. (extensions/telegram/src/polling-session.ts:59, 1db8ab373422)
    • Current source deduplicates stop requests per polling cycle: Normal polling now uses a requestStopForRestart guard plus cached stopPromise and stopBotPromise so the runner and bot stop paths are only started once for a restart request. (extensions/telegram/src/polling-session.ts:1282, 1db8ab373422)
    • Current source resets backoff after healthy polling: Both isolated ingress poll-success and normal polling onPollSuccess call #noteHealthyPollingCycle(), which resets restart attempts and stop-timeout cooldown state. (extensions/telegram/src/polling-session.ts:411, 1db8ab373422)
    • Regression tests cover the fixed behavior: The current tests assert the 30s/10m policy, healthy-cycle reset, isolated ingress stop-timeout cooldown, and repeated stop-timeout cooldown helper behavior. (extensions/telegram/src/polling-session.test.ts:710, 8c802aa68351)
    • Fix provenance from git history: Commit 31828af is titled 'fix(telegram): slow polling restart storms' and changes the old 2s/30s policy to 30s/10m while adding the restart backoff state, stop-timeout cooldown, stop dedupe, and healthy-cycle reset. (extensions/telegram/src/polling-session.ts:53, 31828af1990e)
    • Merged PR provenance: GitHub ties commit 31828af to the merged Telegram polling restart-storm pull request, and the maintainer landing comment records current-head real behavior proof plus merge commit 0b26a1b. (0b26a1bca728)

    Likely related people:

    • TurboTheTurtle: Authored the merged Telegram polling restart-storm fix that directly implements this issue's requested backoff, cooldown, stop dedupe, and reset behavior. (role: fix author; confidence: high; commits: 31828af1990e; files: extensions/telegram/src/polling-session.ts, extensions/telegram/src/polling-session.test.ts)
    • obviyus: Added the final stop/timer refinement commits on the same PR and posted the landing proof for the rebased Telegram polling fix. (role: recent area contributor and lander; confidence: high; commits: 3c935ac53e3d, 20f1d98c63f9, 0b26a1bca728; files: extensions/telegram/src/polling-session.ts, extensions/telegram/src/polling-session.test.ts)
    • Josh Avant: Recent commits before the fix worked on stalled isolated spool handlers and the isolated polling stall watchdog in the same runtime surface. (role: adjacent isolated polling owner; confidence: medium; commits: b7735f88fa27, 40db92f6097c; files: extensions/telegram/src/polling-session.ts)
    • steipete: Recent Telegram polling-session commits before this report touched timer bounds, reconnect delivery draining, and the polling-session extraction history. (role: recent area contributor; confidence: medium; commits: 90d569e896a5, 7270bb95b70e, 1d301f74a6a9; files: extensions/telegram/src/polling-session.ts, extensions/telegram/src/polling-session.test.ts)

    Codex review notes: model internal, reasoning high; reviewed against 1db8ab373422; fix evidence: merged PR #88832, release v2026.6.2-beta.1, commit 31828af1990e.

  2. added
    P1High-priority user-facing bug, regression, or broken workflow.
    impact:crash-loopCrash, hang, restart loop, or process-level availability failure.
    impact:message-lossChannel message delivery can be lost, duplicated, or misrouted.
    issue-rating: 🐚 platinum hermitGood issue quality with a plausible reproduction path needing some confirmation.
    clawsweeper:needs-live-reproClawSweeper needs live local, crabbox, or manual validation to confirm this issue.
    on May 30, 2026
  3. wangpuv commented on Jun 9, 2026

    @wangpuv

    Additional reproduction on v2026.6.1 (macOS, launchd):

    Observed behavior:

    1. Telegram provider restarted itself at 09:10 — auto-restart attempt 1/10 → health-monitor restarting (reason: stopped) → auto-restart attempt 2/10 within seconds
    2. After restart, switched to isolated polling ingress
    3. During this restart window, multiple chat replies were silently dropped — sendMessage log entries for those message IDs are completely absent
    4. User saw "No response generated" on Telegram for messages that the agent did generate but never delivered

    Evidence:

    09:10:34 [telegram] auto-restart attempt 1/10 in 5s
    09:10:41 [telegram] starting provider
    09:10:43 [telegram] auto-restart attempt 2/10 in 10s
    09:10:46 [health-monitor] restarting (reason: stopped)
    09:11:08 [telegram] starting provider
    09:11:38 [telegram] isolated polling ingress started
    

    Messages sent before 09:10 had outbound send ok log. Messages during the restart window have zero sendMessage entries — the replies were generated by the agent but lost in the message pipeline.

    This aligns with both the aggressive restart policy reported in this issue and the in-flight message loss described in #78380.

  4. locked as resolved and limited conversation to collaborators on Jun 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    P1High-priority user-facing bug, regression, or broken workflow.clawsweeper:needs-live-reproClawSweeper needs live local, crabbox, or manual validation to confirm this issue.impact:crash-loopCrash, hang, restart loop, or process-level availability failure.impact:message-lossChannel message delivery can be lost, duplicated, or misrouted.issue-rating: 🐚 platinum hermitGood issue quality with a plausible reproduction path needing some confirmation.

    Type

    No type

    Fields

    Priority

    None yet

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions