Skip to content

autoscale wakes forever for a run that will never start #65

Description

@aicayzer

What happens

A GitHub Actions run can enter queued and stay there permanently with zero jobs attached. _rp_autoscale counts queued runs via total_count, so it sees work, brings the pool up, finds nothing to do, and _rp_sweep stands it down after RUNPOOL_IDLE_SECS. The next tick sees the same queued run and repeats.

The result is a wake every RUNPOOL_IDLE_SECS plus one tick, indefinitely. Observed in the wild: 29 wakes in one day, spaced 21 minutes apart through the night, against a normal five to eight, for two days until the run was cancelled by hand. It had happened at least once before on the same install.

Why nothing reports it

A pool that wakes and stands down is behaving exactly as designed, so status and doctor both read healthy and no notification fires. The only trace is a repeating autoscale: ... bringing up line in the log, which nobody reads unless already suspicious.

Why the obvious fixes are wrong

  • An age rule ("a run queued longer than N hours is stuck") breaks on a laptop that sleeps. Work queued overnight is legitimately hours old and must still be served.
  • Suppressing the pool would blind an org pool to every other repository it watches because of one stuck run in one of them.
  • A fingerprint of the whole queued set never accrues consecutive evidence on a busy org, because the set changes whenever any other run arrives or completes.
  • Counting ticks would punish a pool that cannot start at all: _rp_up returns early on a missing launch agent, so it would take a strike a minute and then refuse real work once repaired.

The fix

Track individual run IDs with a strike count, and compute the wake as queued > held rather than suppressing anything. A strike is earned only when the pool's .started stamp has changed since that run was last judged, which is proof the pool came up and back down. A held run stops counting as work; every other run in every other repository still wakes the pool normally. Suppression clears itself when the run leaves the queued set, with a daily re-arm so a run held by something transient gets re-checked.

Report it through doctor as a warning naming the run, and once through the notifier.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions