scaleset: reap runners stranded by GitHub Actions outages - #841
Open
benoit-nexthop wants to merge 2 commits into
Open
scaleset: reap runners stranded by GitHub Actions outages#841benoit-nexthop wants to merge 2 commits into
benoit-nexthop wants to merge 2 commits into
Conversation
When all runners of a scale set are busy, it is hard to tell where a given workflow job sits in the queue, or why capacity is not being used. This adds a "Job Queue" view to the web UI that groups queued and running jobs per scale set (and per pool, matched by labels), ordered by request time, with links to the GitHub run/job pages and to the scale set/pool detail pages. To support the view: * Jobs recorded by scale set listeners now carry the garm scale set ID (new WorkflowJob.ScaleSetFkID column, exposed as scale_set_id on the Job API model). Previously the scale set ID was dropped when recording jobs. * The full RunnerScaleSetStatistic from each session message is persisted on the scale set and exposed as "statistics" on the API (previously only TotalAssignedJobs survived as desired_runner_count). The view shows GitHub's numbers (assigned jobs, busy/idle runners) next to GARM's instance counts, making divergence visible. * consolidateRunnerState syncs GitHub's per-runner view (online idle/busy, offline) onto instances that finished installing. Without this, a runner whose agent died after setup stayed "idle" in GARM forever while GitHub considered it offline and never assigned it jobs. The view updates live via the existing job/instance websocket events. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
During the 2026-08-06 GitHub Actions outage, hundreds of scale set runners ended up stranded in running/offline: the agent registered, then died or lost its connection to the actions service. GARM only tracked GitHub's offline status as metadata and never recycled such runners, pinning every scale set at max_runners and starving the fleet. Changes: - Track when each runner is first observed in running/offline (in memory on the scale set worker; there is no status-changed timestamp in the DB and UpdatedAt is bumped by every write) and recycle it after DefaultRunnerOfflineTimeout (10 minutes) plus a deterministic per-runner jitter (fnv hash of the name, up to 5 minutes). The jitter prevents runners that went offline together from being reaped, respawned and going offline together in ever more synchronized batches. Offline status is only refreshed by the periodic consolidation pass, so effective reap latency is 10-15+ minutes. - Cap offline reaps at max(5, max_runners/4) per consolidation pass. Provider delete/create operations fan out one goroutine per instance, and unbounded churn can overwhelm the provider API. - When GitHub refuses to deregister a runner (TaskAgentJobStillRunningException, seen when the broker lost a job assignment the runner never acquired), log the refusal and proceed with pending_delete instead of aborting. Destroying the instance is exactly what un-sticks GitHub's state: the agent disappears, GitHub fails the stuck job and releases the runner. Note: the github runners list is NOT a reliable busy indicator for scale set runners (it reports idle for runners actively executing a job), so runners GARM considers active are never touched based on it. With this, scale sets keep cycling stranded runners during an outage so fresh runners can pick up whatever jobs do go through, and the fleet recovers on its own once GitHub stabilizes. Battle-tested in production during the outage itself. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
benoit-nexthop
force-pushed
the
offline-runner-reaper
branch
from
August 7, 2026 06:09
25212ad to
54148cc
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #836 — the first commit here is that PR's status-sync change, which this depends on (it is what surfaces GitHub's offline/idle/busy view onto instances during consolidation). Once #836 merges and this branch is rebased, the diff collapses to the single reaper commit.
During the 2026-08-06 GitHub Actions outage, hundreds of scale set runners ended up stranded in running/offline: the agent registered, then died or lost its connection to the actions service. GARM only tracked GitHub's offline status as metadata and never recycled such runners, pinning every scale set at max_runners and starving the fleet.
Changes:
DefaultRunnerOfflineTimeout(10 minutes) plus a deterministic per-runner jitter (fnv hash of the name, up to 5 minutes). The jitter prevents runners that went offline together from being reaped, respawned and going offline together in ever more synchronized batches. Offline status is only refreshed by the periodic consolidation pass, so effective reap latency is 10-15+ minutes.max(5, max_runners/4)per consolidation pass. Provider delete/create operations fan out one goroutine per instance, and unbounded churn can overwhelm the provider API.TaskAgentJobStillRunningException, seen when the broker lost a job assignment the runner never acquired), log the refusal and proceed with pending_delete instead of aborting. Destroying the instance is exactly what un-sticks GitHub's state: the agent disappears, GitHub fails the stuck job and releases the runner.Caveat learned in production: the GitHub runners list is not a reliable busy indicator for scale set runners — it reports idle for runners that are actively executing a job. Runners GARM considers active are therefore never touched based on the list (an earlier iteration of this patch did, and cancelled running jobs).
With this, scale sets keep cycling stranded runners during an outage so fresh runners can pick up whatever jobs do go through, and the fleet recovers on its own once GitHub stabilizes. Battle-tested in production during the outage itself: ~340 stranded runners across 5 scale sets were recycled automatically and the fleet tracked GitHub's recovery with no human intervention.
🤖 Generated with Claude Code