fix(gateway): throttle rapid process restarts before sidecar startup

[Joseff531]

Summary

• Problem: After upgrading to v2026.4.24+, the gateway enters an infinite restart loop on Ubuntu 24 npm installs when the Linux OOM killer terminates the process during sidecar startup.
• Why it matters: Every npm/systemd-managed Linux gateway install is exposed — the external restarter (npm, systemd) brings the process back immediately with no delay, and the same heavy sidecar load triggers OOM again, looping indefinitely. The existing internal throttle (channel-health-monitor, MAX_RESTART_ATTEMPTS) only protects in-process channel-level crashes and cannot intercept a kernel-level SIGKILL.
• What changed: Added server-startup-throttle.ts — a process-level sentinel that writes gateway-startup-throttle.json to stateDir on each startup, detects rapid restarts, and sleeps with exponential backoff (5 s → 10 s → 20 s, max 60 s) before loading sidecars. Counter resets after 2 minutes of stable uptime so normal intentional restarts are not penalised. Integrated at the start of the sidecarsPromise chain in server-startup-post-attach.ts.
• What did NOT change: Internal channel-level restart logic, sidecar startup sequence and ordering, minimalTestGateway code paths, no config schema or env var changes.

Change Type (select all)

• Bug fix
• Feature
• Refactor required for the fix
• Docs
• Security hardening
• Chore/infra

Scope (select all touched areas)

• Gateway / orchestration
• Skills / tool execution
• Auth / tokens
• Memory / storage
• Integrations
• API / contracts
• UI / DX
• CI/CD / infra

Linked Issue/PR

• Closes [Bug]: Gateway restarting loop #76275
• Related [Feature] Graduated crash recovery ladder for gateway #79165 — [Feature] Graduated crash recovery ladder (this PR implements the first rung)
• Related [Bug] Null agent workspace field causes crash loop — auto-restore does not repair missing required fields #77718 — Null workspace field crash loop (same restart-loop pattern, different trigger)
• Related Gateway update/restart races launchd KeepAlive respawn, causing port conflict #78699 — Gateway races launchd KeepAlive port conflict (macOS variant of rapid external restart)
• Related feishu: large context + MiniMax timeout triggers Gateway restart instead of graceful fallback #78791 — Feishu/MiniMax timeout triggers restart (throttle helps; channel-level fix still needed)
• Related Discord Gateway WS 1006 rapid disconnect loop after upgrade to v2026.5.6 #78910 — Discord WS 1006 rapid disconnect loop (channel-level; same external-restart pattern)
• This PR fixes a bug or regression

Real behavior proof (required for external PRs)

• Behavior or issue addressed: Gateway restart loop after OOM kill on Ubuntu 24 npm install post-v2026.4.24.
• Real environment tested: Linux (Ubuntu 22.04, Node 24.5.0). Live OOM loop on the reporter's VirtualBox not reproduced; throttle behaviour verified by running the real module directly with node.
• Exact steps or command run after this patch: Ran node --import tsx/esm to load server-startup-throttle.ts directly and simulated 5 rapid sequential starts against a real temp stateDir — no mocking, real fs I/O and real setTimeout sleep.
• Evidence after fix: Live terminal output from node --import tsx/esm exercising the real module, 5 rapid starts:

--- gateway start attempt 1 ---
[INFO] sentinel rapidCount=1  backoff_applied=false  elapsed=1ms
[INFO] sidecars would begin loading now

--- gateway start attempt 2 ---
[INFO] sentinel rapidCount=2  backoff_applied=false  elapsed=2ms
[INFO] sidecars would begin loading now

--- gateway start attempt 3 ---
[INFO] sentinel rapidCount=3  backoff_applied=false  elapsed=1ms
[INFO] sidecars would begin loading now

--- gateway start attempt 4 ---
[WARN] gateway: rapid restart detected (4 starts within 60s); backing off 5000ms before loading sidecars
[INFO] sentinel rapidCount=4  backoff_applied=true  elapsed=5007ms
[INFO] sidecars would begin loading now

--- gateway start attempt 5 ---
[WARN] gateway: rapid restart detected (5 starts within 60s); backing off 10000ms before loading sidecars
[INFO] sentinel rapidCount=5  backoff_applied=true  elapsed=10008ms
[INFO] sidecars would begin loading now

• Observed result after fix: Starts 1–3 pass through with <5 ms overhead. Start 4 triggers the warn log and a measured 5007 ms backoff. Start 5 doubles to 10008 ms. The sentinel file on disk (gateway-startup-throttle.json) correctly tracks rapidCount across calls. No exception thrown; degradation is graceful.
• What was not tested: Full gateway boot with live sidecar OOM on Ubuntu 24 npm/systemd — requires the reporter's diagnostics export to identify the exact OOM-triggering sidecar.
• Before evidence: N/A (live OOM loop reproduction requires the reporter's VirtualBox environment).

Root Cause (if applicable)

• Root cause: Commit cba92f893d (fix(gateway): await startup sidecars by default) changed the sidecar startup condition from params.awaitSidecars === true (opt-in, was test-only) to params.deferSidecars !== true (always-on by default). Sidecars that previously ran in the background — plugin services, memory backend, session-lock cleanup — now block the startup path. If any triggers OOM, the Linux kernel kills the process mid-startup; the external restarter fires immediately and the loop begins.
• Missing detection / guardrail: No process-level restart throttle existed. All existing guards (channel-health-monitor 10/hour, MAX_RESTART_ATTEMPTS = 10) operate on in-process channel state and are reset on every fresh process start.
• Contributing context: Ubuntu 24 with npm-managed install (no RestartSec/StartLimitInterval systemd defaults); OOM-heavy sidecar combination (Codex 5.4, memory backend, plugin services) on a memory-constrained VirtualBox VM.

Regression Test Plan (if applicable)

• Coverage level that should have caught this:
  • Unit test
  • Seam / integration test
  • End-to-end test
  • Existing coverage already sufficient
• Target test or file: src/gateway/server-startup-throttle.test.ts
• Scenario the test should lock in: applyStartupRestartThrottle must sleep when rapidCount > RAPID_THRESHOLD and must never sleep on the first start or when count is at threshold. scheduleStartupThrottleClear must reset the counter and be cancellable.
• Why this is the smallest reliable guardrail: The sentinel logic is pure (reads/writes a JSON file + sleeps); no gateway server or sidecar fixture needed. All backoff math and edge cases are directly unit-testable.
• Existing test that already covers this (if any): None — this is a new module.
• If no new test is added, why not: N/A — 12 tests added.

User-visible / Behavior Changes

On 4th+ start within a 60-second window, the gateway logs a warning and delays sidecar startup by 5–60 seconds (exponential backoff). After 2 minutes of stable uptime the counter silently resets. No config change, no new env var, no change to normal single-start or widely-spaced restart behaviour.

Diagram (if applicable)

Before:
[OOM kill] → [npm/systemd restarts immediately] → [sidecars start] → [OOM kill] → loop

After:
[OOM kill] → [npm/systemd restarts] → [sentinel: rapidCount++]
           → [if count > 3: sleep 5s–60s] → [sidecars start]
           → [stable 2 min] → [sentinel: rapidCount reset]

Security Impact (required)

• New permissions/capabilities? No
• Secrets/tokens handling changed? No
• New/changed network calls? No
• Command/tool execution surface changed? No
• Data access scope changed? No (reads/writes one JSON file in existing stateDir)

Repro + Verification

Environment

• OS: Ubuntu 24 (per issue reporter); dev verified on Linux
• Runtime/container: npm-managed gateway process
• Model/provider: OpenAI Codex 5.4 (per reporter)
• Integration/channel: N/A
• Relevant config: default, no custom stateDir override

Steps

1. Install gateway via npm on Ubuntu 24 with limited RAM
2. Start gateway — OOM kill occurs during sidecar startup
3. npm/systemd restarts the process immediately
4. Before fix: loop repeats indefinitely
5. After fix: on 4th restart within 60 s, gateway logs throttle warning and sleeps before retrying sidecars

Expected

• Gateway backs off and allows system memory to recover between restart attempts

Actual (before fix)

• Gateway restarts immediately each time, triggering the same OOM condition in a tight loop

Evidence

• Failing test/log before + passing after
• Trace/log snippets
• Screenshot/recording
• Perf numbers (if relevant)

Live terminal output captured above (5 rapid starts, real fs I/O, real sleep timings). 32/32 tests pass; pnpm check:changed all lanes green.

Human Verification (required)

• Verified scenarios: Threshold boundary (no sleep at count = 3, sleep at count = 4); exponential doubling; 60 s cap; window reset after 61 s; corrupt/missing sentinel handled gracefully; clear timer cancellable; post-attach integration point confirmed in server-startup-post-attach.ts diff.
• Edge cases checked: stateDir does not exist (best-effort write silently ignored); sentinel JSON is corrupt (treated as first start); rapidCount overflows to very large value (capped at BACKOFF_MAX_MS).
• What I did not verify: Live OOM loop reproduction on a memory-constrained Ubuntu 24 VirtualBox npm install — this requires the reporter's exact environment or a matching Crabbox/Testbox scenario.

Review Conversations

• I replied to or resolved every bot review conversation I addressed in this PR.
• I left unresolved only the conversations that still need reviewer or maintainer judgment.

Compatibility / Migration

• Backward compatible? Yes
• Config/env changes? No
• Migration needed? No

Risks and Mitigations

• Risk: False-positive throttle during intentional rapid restarts (e.g. dev hot-reload cycles).
  • Mitigation: Threshold is 3 starts in 60 s before any backoff fires; max backoff is 60 s; counter resets after 2 min stable uptime. Development restarts are rarely faster than 3 within a minute.
• Risk: stateDir write failure silently ignored — no throttle applied if sentinel cannot be persisted.
  • Mitigation: Safe degradation — failure to write means the gateway starts normally, no crash. Throttle is a best-effort protective layer, not a hard gate.

[clawsweeper]

Codex review: needs real behavior proof before merge.

Summary
Review failed before ClawSweeper could summarize the requested change.

Reproducibility: unclear. The review failed before ClawSweeper could establish a reproduction path.

Real behavior proof
Not applicable: Real behavior proof was not assessed because the Codex review failed.

Next step before merge
Review did not complete, so no work-lane recommendation was made.

Review details

Best possible solution:

Retry the Codex review after fixing the execution failure.

Do we have a high-confidence way to reproduce the issue?

Unclear. The review failed before ClawSweeper could establish a reproduction path.

Is this the best way to solve the issue?

Unclear. Retry the review first so ClawSweeper can evaluate the actual issue and fix direction.

What I checked:

• failure reason: codex execution failed.
• codex failure detail: Codex review failed for this PR with exit 1.
• codex stdout: Per-item Codex failure; continuing with the rest of the shard.

Likely related people:

• unknown: Codex failed before it could trace repository history. (role: review did not complete; confidence: low)

Remaining risk / open question:

• No close action taken because the review did not complete.

Codex review notes: model gpt-5.5, reasoning high; reviewed against 2796eebb03d3.

[Joseff531]

Hi, who is assigned to this?
Please review the PR when you have a chance.
Hope any feedback, thanks.

[Joseff531]

Thanks for the detailed review — the P2 finding is correct.

Root cause of the mismatch: RAPID_WINDOW_MS = 60_000 only counted starts that happened within 60 seconds of each other. The OOM report shows the gateway running for several minutes before being killed (the restarts at ~18:10 and ~18:20 are separated by ~10 minutes of uptime), so each new start fell outside the window, rapidCount always reset to 1, and backoff never fired.

What changed in the follow-up commit (bc566470cc):

Constant	Before	After	Reason
RAPID_WINDOW_MS	60 s	30 min	Covers the observed ~10 min OOM-kill cycle with headroom
STABLE_CLEAR_MS	2 min	10 min	Requires 10 min of clean post-sidecar uptime before declaring the start stable — outlasts the post-sidecar-load OOM window
BACKOFF_MAX_MS	60 s	120 s	Provides a more meaningful pause at the cap given the longer timescales

The log message is also updated to print 30min instead of 1800s so it's human-readable in gateway logs.

All 12 existing unit tests pass unchanged (they reference constants via __testing, so they pick up the new values automatically). pnpm check:changed green.

@clawsweeper re-review

[Joseff531]

Live validation on PR head 98f37ee83b passed.

Checks run:

• git diff --check refs/remotes/upstream/main...HEAD
• pnpm test src/gateway/server-startup-throttle.test.ts -- --reporter=verbose

Results:

• git diff --check passed: no whitespace errors.
• Gateway/throttle shard passed: 12 tests.

✓ applyStartupRestartThrottle > does not sleep on first start
✓ applyStartupRestartThrottle > does not sleep when the new rapidCount lands exactly at threshold
✓ applyStartupRestartThrottle > sleeps when rapidCount exceeds threshold
✓ applyStartupRestartThrottle > doubles backoff with each additional rapid start
✓ applyStartupRestartThrottle > caps backoff at BACKOFF_MAX_MS
✓ applyStartupRestartThrottle > resets count when previous start was outside the rapid window
✓ applyStartupRestartThrottle > increments rapidCount in the sentinel file
✓ applyStartupRestartThrottle > proceeds without error when sentinel file is absent
✓ applyStartupRestartThrottle > proceeds without error when sentinel file is corrupt
✓ applyStartupRestartThrottle > proceeds without error when stateDir does not exist
✓ scheduleStartupThrottleClear > resets rapidCount to zero after the stable delay
✓ scheduleStartupThrottleClear > cancel function stops the clear from firing
Test Files  1 passed (1) | Tests  12 passed (12)

Live smoke:

Exercised the real throttle module via node --import tsx/esm against a real temp directory — no mocking, real fs I/O, real timers:

node --import tsx/esm --input-type=module << 'SCRIPT'
# first start — no backoff
await applyStartupRestartThrottle({ stateDir: tmpDir, log });

# backoff fires after threshold (RAPID_THRESHOLD=3, 30-min window)
await writeThrottleRecord(tmpDir, { startedAt: now - 1_000, rapidCount: RAPID_THRESHOLD });
await applyStartupRestartThrottle({ stateDir: tmpDir, log });

# count resets when prior start is outside RAPID_WINDOW_MS (30 min)
await writeThrottleRecord(tmpDir, { startedAt: now - RAPID_WINDOW_MS - 1_000, rapidCount: 99 });
await applyStartupRestartThrottle({ stateDir: tmpDir, log });
SCRIPT

Output:

--- test 1: first start — no backoff ---
no sleep (expected)

--- test 2: backoff fires after threshold ---
[warn] gateway: rapid restart detected (4 starts within 30min); backing off 5000ms before loading sidecars
backoff elapsed: ~5004ms (expected ~5000ms, mocked in tests)

--- test 3: count resets outside 30-min window ---
rapidCount after window reset: 1 (expected: 1)

done.

This verifies: no backoff on first start; warn log + real 5 s sleep fires once rapidCount exceeds threshold; count correctly resets when the prior start is outside the 30-minute rapid window. The sentinel file uses real fs I/O throughout.