Skip to content

v2.4.0 — Runner self-update mechanism

Choose a tag to compare

@jhuebel jhuebel released this 17 Jul 21:05
· 3 commits to main since this release

Runner self-update mechanism

Runners can now check in with the management plane for newer released builds and update themselves in place, closing the last piece of fleet operations that still required manually rebuilding, copying, and restarting each runner by hand.

Why

As the runner fleet grows, keeping every host's binary current is real toil, and until now there was no visibility into which runners were running stale code beyond the version shown in Settings → Runners. This reuses the existing 15s heartbeat as the discovery channel and the runner's existing Sanctum bearer token for the download — no new polling loop, no new credential type.

What's new

  • Self-update over heartbeat — if Settings → Runners' "Automatically offer runner updates" toggle is on, a heartbeat response includes an update field whenever a newer released build exists for that runner's os/arch. The runner downloads it, verifies its sha256 checksum, replaces its own binary in place (atomic rename on Linux; rename-aside-then-write on Windows), and exits cleanly so the process supervisor (systemd/NSSM) relaunches it — never interrupting a script that's currently running.
  • Two-step publish → release, always — automator:publish-runner-binary stores a binary as a draft (checksum + size computed, never offered to the fleet). Only a separate, explicit automator:release-runner-binary makes it live. A bad build can never reach the fleet automatically and unattended. packaging/build.sh now publishes both cross-compiled binaries on every build, but never auto-releases.
  • Rollback is just re-releasing — releases are ordered by released_at, not semver, so shipping an older version again is the entire rollback procedure.
  • Settings visibility — the global toggle and a per-runner "(update to X.Y.Z available)" hint (shown independent of the toggle, so staleness is visible even with auto-update off) were added to Settings → Runners.
  • New automator-runner version subcommand for standalone on-host debugging.

Verified end-to-end against a live runner: idle-gate deferral while a job is running, successful update + supervisor restart once the job finishes, checksum-mismatch refusal leaving the running binary untouched, and rollback via re-release. The Windows replace/restart path is implemented and unit-tested but hasn't been exercised on a live Windows host yet.

Upgrading

One new table (runner_releases) and one new column (runner_auto_update_enabled on app_settings) — run php artisan migrate --force as usual. Auto-update is off by default; nothing changes for existing runners until you publish a build, release it, and turn the toggle on.