fix: back off repeated GitHub CLI reads - #6345
Merged
Merged
Conversation
…y tick A gh blip during a real incident let branch-reconcile and updateChecker re-fire the same 60s-timeout `gh pr list` / release-check call for every managed repo on every scheduler tick with no cooldown, piling up concurrent gh processes while gh was already struggling (confirmed in PM2 logs: repeated timeouts across PortOS, gradient-play, and BookLoom within a few minutes). ensureForgeReachable's 60s health probe doesn't catch this because it's a separate, cheaper call that can pass while the heavier read still hangs. Add an opt-in `backoffKey` to execGh() with exponential backoff per key, cleared on success. Mutating/one-off gh calls are unaffected (they don't pass a key). Wired into branchReconcile's PR-list read and updateChecker's release check — the two call sites that hit this in the observed incident. Filed #6343 to extend it to the remaining periodic gh readers (prWatcher, issueReconcile, blockedIssueReconcile).
The backoff added in the previous commit is keyed the same whether
checkForUpdate() is called by the unattended 30-min scheduler or by a
user clicking "check for updates" (POST /api/update/check). Without this,
a cooldown armed by the background poller's own failure would silently
reject an explicit user-triggered check for up to 15 minutes with no real
attempt. checkForUpdate() now takes { manual: true } to skip the
backoffKey; the route passes it, the scheduler doesn't.
Owner
Author
|
Required code review was not completed before publication. This PR is intentionally left open and will not be merged until the required review completes. |
This was referenced Sep 5, 2026
atomantic
added a commit
that referenced
this pull request
Sep 5, 2026
updateChecker's 30-min scheduler interval always exceeds the 15-min backoff cap #6345 hardcoded, so the cooldown from a failed release check expires before every subsequent tick and never actually suppresses a retry -- a silent no-op for its only production caller. execGh now takes an optional backoffMaxMs override; branch-reconcile and the other pollers keep the default (appropriate for their much faster tick rate), and updateChecker passes 3x its own interval so repeated failures actually widen the gap between real attempts.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes #6343
Validation
npm test -- services/github.test.js services/branchReconcile.test.js services/updateChecker.test.js services/prWatcher.test.js services/issueReconcile.test.js services/blockedIssueReconcile.test.js services/taskDataInputs.test.js— 418 passednpm test— 39,846 passed, 35 skippedReview gate
agy --effort mediumreviewer found and the fix was applied for one low-severity child lifecycle finding.review-blocked. Per delivery policy this PR is left open for cold pickup and is not being merged.