v0.3.0 — More than one Mac, one verdict, Fleet Cockpit

Full changelog: v0.2.0...v0.3.0 · Upgrading from an earlier version: one-time steps · Docs: addisdev.github.io/actions-runners

Thanks to @wized2 for the inferred-grouping tests (#22), the first contribution from outside the project.

More than one Mac, and one answer for the whole fleet. A second host joins in
one command, standby tiers run CI on a primary host first, and a switch takes a
host out of rotation. The fleet verdict reduces everything the dashboard knows
to one ordered answer, and Fleet Cockpit, its CLI and MCP server carry it to
the menu bar and to agent sessions. The dashboard also works away from the
desk: LAN and Tailscale access, per-device pairing, and a mobile layout. The
autofix bridge is removed.

Added

• Host on/off switch. A switch beside each host's name in the Fleet and
  Hosts tabs (on a single host, above the runner grid) takes the whole host
  out of CI: every runner drains with owner host, busy ones after their
  current job, and switching back on resumes only those. The confirm names the
  jobs still finishing and any repo no other host serves. host.off/host.on
  actions (agents too; cockpit run --host), scripts/host-drain.sh --off/--on/--status,
  host marker first line off. An off host is unusable to tiers and
  placement, refuses registration, and gets nothing from agent autonomy.
  /api/hosts gains off, drainingCount, activeCount and impact;
  /api/state gains hostImpact. The old host drain is now the secondary
  Pause new runners. See
  Taking a host out of rotation.

• Jobs orphaned by a draining runner are named, not waited on. A job that
  passed every step but whose runner went away during "Complete runner" stays
  in_progress on GitHub until someone force-cancels the run (addisdev/lake-api
  PR #73, 2026-10-12/13: three jobs, 10-25 minutes each, while cockpit why said
  "0 of 1 done" and cockpit wait kept waiting). The cockpit now asks GitHub
  for the jobs of runs active past three minutes; a job whose work steps have
  all completed, stuck at a "Complete …" step for three minutes, with its runner
  draining, offline, dead or lost (or with "Complete runner" cancelled, which
  only a runner shutdown does) is reported as orphaned by a draining runner —
  not your code, with the force-cancel and gh run rerun commands.
  cockpit why lists it; cockpit wait and wait_for_checks stop with exit
  2; cockpit wait --pr N --recover force-cancels and re-runs that PR's own
  orphaned runs once each (never a run whose work failed), logs each to
  ~/Library/Logs/FleetCockpit/orphan-recover.ndjson, and keeps waiting.
  CockpitCore/OrphanedJob.swift.

• A full shared memory table is now a verdict, not a mystery red test.
  hooks/shm-reap.sh logged orphan sweeps and low events to
  dashboard/logs/shm.ndjson, and nothing read it; the shmget error lives in
  step logs, which no failure class reads, so cockpit why called those
  failures ordinary. fleetd.js and agent.js now sample ipcs -m -o -p each
  tick (orphans the next hook will remove count as free) and summarise the log
  tail into the host vitals (host.shm, glance shm* vitals). A new shm-low
  rung, between dead-service and saturated, opens when fewer than
  FLEET_SHM_LOW_FREE segments are free on any reporting host or a low event
  is under 30 minutes old (critical when the table is full), with an alert of
  the same name. Host cards show free segments and orphans reaped in 24 h, the
  leak rate. The cockpit decodes the rung, the vitals and the history lane, and
  cockpit why flags it beside a repo's failures. dashboard/lib/shm.js.

• Standby tiers. Run CI on a primary host first and bring a standby
  host's runners online only when the primary is at its admission cap for 60 s,
  has a run queued over 2 min with no idle matching runner, or is unfit
  (disk-floor, saturated, a dead primary runner, critical memory pressure,
  or a stale heartbeat); drain them again after 10 min of headroom and an empty
  queue. A floor set of standby runners stays online always. While a primary
  lane is at cap its idle runners are drained too, so GitHub stops assigning
  it jobs it would only hold, but never without an online standby twin.
  dashboard/lib/tiers.js, wired into the fast tick, FLEET_TIERS_*
  (off/observe/enforce), shown as tiers on /api/state and the glance.
  See Standby tiers.

• drain-runner.sh --by=<owner> records who drained a runner, and
  --resume --by=<owner> only clears that owner's marker, so the controller
  never resumes an operator's drain. drain-stop-when-idle.sh keeps the owner.

• Agent: tiers.drain/tiers.resume batch commands, an optional
  FLEET_AGENT_COMMANDS allowlist (drain/resume only), and the autonomy rule:
  after FLEET_AGENT_AUTONOMY_S (180 s) without a heartbeat it resumes the
  runners the controller drained on its host.

• scripts/join-host.sh preflight also checks the Xcode license (xcodebuild -version answers without it, then swiftc exits 69 in the first job), an
  optional FLEET_XCODE_VERSION every host must match (a second host with an
  older Xcode builds the same job against a different SDK), and PyYAML for both
  python3 interpreters a job can reach. All three were hit adding a second
  host.

• Preflight checks plain CLI tools. preflight.sh checked Xcode, Postgres,
  Java, node and gh but nothing a Makefile calls, so a repo's make lint met
  make: shellcheck: No such file or directory on a new host (2026-10-09).
  scripts/cli-tools.txt lists the tools (shellcheck, jq, make always; deno
  and ruby when a self-hosted workflow runs them, via infer-checks.py), and
  scripts/check-tools.sh looks them up on the runners' PATH. join-host.sh
  checks every listed tool and --install-tools brew-installs the missing
  ones. Preflight also warns when the runners' python3 is PEP 668 externally
  managed. Tests: scripts/test-preflight-tools.sh.

• Admission decisions from every host. An agent host's hooks log locally
  like the coordinator's, and nothing read that file, so holds on a second host
  never reached the dashboard or the admission-hold alert. The agent now ships
  the lines added since its last accepted heartbeat (lib/admission-ship.js,
  at-least-once, offset advanced only by the coordinator's count), and the
  coordinator stores them by host: admission_events.host_id, waiting[].host
  and admission.byHost. The headline mode and limit stay the coordinator's
  own. host_samples.host_id records which machine a vitals row describes, so
  moving the coordinator does not splice two machines' load into one history.
  Runbook: docs/federation.md, "Moving the coordinator to another Mac".

• One-command second host. scripts/join-host.sh joins a Mac to the fleet:
  preflight (including the toolchains jobs use), the agent, mirrored runners and
  the health timer, dry run unless --apply. scripts/mirror-runners.sh and
  dashboard/lib/mirror.js copy the coordinator's runners one per distinct
  label set, skipping what the host cannot run (a missing label, Simulator
  runners without Xcode, FLEET_MIRROR_SKIP_REPOS), with registration tokens
  optionally minted elsewhere (--tokens-from). Template:
  examples/fleet.env.second-host. The agent and installer accept the token
  from FLEET_AGENT_TOKEN_FILE. Runner pin 2.336.0 → 2.337.0.

• Fleet verdict. lib/verdict.js reduces drift, queue causes, failure
  classes, admission holds and alerts to one ordered ladder — cannot read
  GitHub, host down, disk floor, dead service, saturated, account blocked,
  config drift, waiting, clear — with evidence and one next move, plus a state
  for every runner. Published as verdict on /api/state. Checked against
  recorded incidents in test/fixtures/scenarios.js.

• GET /api/glance and /api/stream?view=glance: a versioned
  (schema: 1) ~10 KB payload for small screens and slow links, where the full
  state is ~130 KB.

• admission-hold alert. A job held by the admission disk floor opens a
  critical alert after 2 minutes — previously jobs sat at "Set up runner" with
  no error anywhere. A slot wait past FLEET_ADMIT_MAX_WAIT_S opens a warning.

• host-saturated alert. Two runner-lost jobs inside an hour — the
  signature of a starved host — open a warning while it is happening.

• scripts/scrub-snapshot.mjs turns a live snapshot into a committable fixture.

• Fleet Cockpit (cockpit/), a macOS menu bar client of /api/glance:
  verdict, per-host vitals with the disk floor drawn on the gauge, a pill per
  runner, the queue. Reaches a loopback-only dashboard through an SSH tunnel
  that dies with the app; reconnects on wake and network change; keeps a stale
  view greyed rather than green. Bundled incident fixtures, --render for
  screenshots, a cockpit CLI, and a snapshot file for other tools. See
  docs/cockpit/.

• scripts/make-glance-fixtures.mjs keeps the cockpit's fixtures in step with
  the daemon (checked in CI).

• Out-of-band sentinel in the cockpit: when the dashboard does not answer it
  reads SSH reachability, GitHub's view of the host's runners, a runner on
  another machine and githubstatus.com against one table — host asleep, power
  or network out, rebooted with nobody logged in, every service dead, dashboard
  down with the fleet working, this Mac offline — and notifies on transitions.
  cockpit sentinel runs the same check from a terminal.

• The cockpit's Why? ladder, Copy brief (markdown incident brief, also
  cockpit brief), and **Failed in t...

v0.2.0 — Documentation site, two-replica HA, host-first controls

Full changelog: v0.1.0...v0.2.0 · Docs: addisdev.github.io/actions-runners

The documentation release, and the control plane grown up. A handbook site with
concepts, design notes, a scripts reference and screenshots; optional
two-replica HA on managed PostgreSQL; host-first fleet controls; per-role
capacity planning and burst-aware autoscaling; and a dismiss button, because
writing the alerts page down made it obvious that the fleet had no way to say
"yes, I know" about a condition that is real, understood and not going to
change.

Added

• Two-replica control-plane HA. Optional managed PostgreSQL state, advisory
  lock leader election, shared snapshots/commands/settings/cooldowns, standby
  dashboard serving, automatic promotion, and agent endpoint failover.
• Host-first fleet controls. Fleet and Runs identify the owning Mac,
  Capacity separates local and fleet headroom, Hosts shows live vitals and
  remote drain/restart/repair/removal actions, and agent credentials can be
  scoped per host.
• Complete live queue discovery and per-role capacity planning. The fast
  collector now paginates queued and in-progress runs, sizes ci and ui-web
  pools independently under one repo-wide cap, and distinguishes a known
  role-unserved condition from an unsafe arbitrary label mismatch.
• Burst-aware autoscaling and evidence-gated pre-warming. Burst thresholds
  remain opt-in; pre-warming remains off until its forecast precision/recall
  gate passes and every placement, cooldown and active-work guard succeeds.
• Long-running job detection, queue-cause history, fleet-wide placement
  visibility, autofix status, and a Prometheus /metrics endpoint.
• Safe database backup/restore commands and healthctl.sh, which installs a
  non-overlapping launchd repair sweep without modifying GitHub runner plists.
• Hardened admission control. Enforce mode no longer fails open on mutex
  contention, cancellation can fall back to GitHub's API through curl, and
  concurrent hook tests are isolated and deterministic.
• Remediation safety gates. Autofix pauses on stale collector state, honours
  minimum run age, classifies timed-out jobs, writes state atomically, and treats
  mixed infrastructure/code failures conservatively.
• A first runner for a repo that has none, off by default as
  provisionUnserved. Everything the autoscaler did until now was a duplicate,
  built by copying a sibling's labels, so the one queue cause the fleet could
  name and not fix was the repo with no sibling to copy — unserved, which the
  classifier has reported and recommended "register a runner for this repo" for
  since it was written. It exists so the fleet can be trimmed: the host limit
  is 32 runners and the fleet reached 42, but the runners over that line belong
  to real repos, so trimming otherwise means choosing which repos silently lose
  CI. Now a trimmed repo gets a runner back the next time it asks for one.
  It provisions only for work queued right now (every trimmed repo still has
  months of history, and sizing off history would undo the trim in one sweep),
  only for jobs that asked for a self-hosted runner (all ten unserved repos
  on this fleet build on GitHub-hosted ones, where having no runner is correct
  and permanent), and without the headroom gate, which runner.register already
  waives for instance 1 on the grounds that a first runner decides whether a repo
  can build at all rather than how many of its jobs may run at once. Proven end
  to end by removing an example backend's only runner, pushing a build, and
  watching the fleet register a replacement and run the job. The argument is in
  Giving a repo its first runner.
• Queue diagnosis reads one job's labels, not every job's at once. A run's
  jobs each carry their own runs-on:, and the diagnosis was merging them.
  An example Android repo has a branch whose build runs on ubuntu-latest and
  whose instrumentation runs on [self-hosted, macOS]; merged, that became
  [self-hosted, macOS, ubuntu-latest] — a set no machine can carry, which
  slipped past the GitHub-hosted check because it contains self-hosted and
  then failed the label check, reporting a critical mismatch against a
  workflow that is correct. Autoscale read the same merged set, so the runner it
  wanted to register would have advertised ubuntu-latest. Diagnosis now uses a
  single queued job's labels, preferring a self-hosted one, and ignores jobs that
  have already found a runner.
• A github-hosted queue cause. A job whose runs-on: names a GitHub-hosted
  image was being diagnosed as a critical label-mismatch against this
  fleet's runners — "Job needs labels [ubuntu-latest] / Runners carry
  [self-hosted, macOS, ARM64]", which is accurate, and a recommendation to go and
  reconcile those labels, which would mean editing a workflow that is behaving
  correctly. It is now its own cause, decided before the repo's runners are
  considered because the answer does not depend on them, and it recommends the
  three places the answer actually is: minutes, spending limits, and GitHub's
  status page. It is also what keeps first-runner provisioning away from the ten
  repos here that build on GitHub-hosted runners and should.
• An × on every alert row. Clicking it stops that condition notifying
  until it clears. No reason to type, no duration to pick, nothing to expire —
  the condition going away is the expiry, which is what keeps the whole
  feature to a two-column dismissals table. What it deliberately does not do
  is close the alert, which would be a notification loop, since the next tick
  would find the condition still true and open it again; or hide it from
  GET /api/alerts, which would stop the autofix bridge repairing it and
  silently reset its retry budget. Dismissals are recorded against the alert's
  scope rather than its key, so dismissing a failing workflow is not undone by
  the next push minting a new run id. POST /api/alerts/dismiss and /restore
  take no token from the local machine — they change what the dashboard tells
  you, not the fleet, and a control whose point is being one click cannot open
  with "run this shell command and paste the output" — and a Dismissed panel
  that is always visible,
  because with no expiry that visibility is the only thing between a dismissal
  and a fault hidden for ever. The argument is in
  Dismissing alerts.
• First unit tests for the alert engine (dashboard/test/alerts.test.js),
  covering the transition, restart and storm behaviour that had been asserted
  only in comments until something needed to layer on top of it.
• A documentation site at
  addisdev.github.io/actions-runners,
  built with MkDocs Material and published from main. mkdocs build --strict
  and a lychee link check gate every pull request that touches docs/.
• Concepts (docs/concepts.md): what a runner is here, what launchd will
  and will not do for it, drift, why a job is queued, admission against
  autoscaling, inferred groups, and why alerts fire on transitions. The
  handbook had no page that introduced any of this.
• Design notes (docs/design/): dashboard-internals.md split into seven
  arguments, each one grounded in something that went wrong on a real fleet.
• A scripts reference (docs/reference/scripts.md): every script, every
  flag read out of the parsing code, and what each one refuses to do. Nine
  scripts had reached the tree undocumented, and several flags the README
  described did not behave as described.
• Figures, rendered from source in docs/figures/ by docs/tools/:
  architecture, the drift matrix, the queue-cause paths, the daemon's loops,
  federation, a banner and a social card.
• Screenshots of every dashboard tab, captured from a fixture fleet by
  docs/tools/shoot-dash.mjs. The repository previously contained no image of
  any kind.
• docs/brand.md, recording the palette, the mark, the type pairing and how
  every asset is produced.

Changed

• Three alert counts now report what is speaking rather than what is true.
  The header badge, watch/fleet-watch.mjs's fault signature and the autofix
  bridge's storm threshold all counted every open alert. Left alone, four
  dismissed conditions would sit at four of the bridge's six permanently, so two
  real failures would disable all remediation — and the badge would disagree
  with the page it links to. Dismissed conditions still appear in
  GET /api/alerts and are still repaired.
• One name. The dashboard tab title, the web manifest, the package
  description and the mark's own <title> said "Runner Fleet"; the repository
  and the README said other things. Everything now says Actions Runners.
  fleetd, fleetctl.sh and fleet.env are unchanged — those are a daemon
  and its files, not the product.
• docs/architecture.md carries a rendered figure instead of two Mermaid
  blocks.
• docs/installation.md became docs/getting-started.md, ending on a job
  running rather than on a curl. The old URL redirects.
• scripts/check-docs.sh no longer exempts any file from the link check, and
  derives the list of scripts that must be documented from the reference page
  rather than from a hand-maintained array.

Fixed

• Every labelled scale-up was refused, and would have been the first time
  anyone turned autoscaling on. FLEET_HOST_LABELS was empty, and placement
  requires a host to advertise every label a new runner needs, so a host that has
  served ci, ui-web, postgres and xcode-26 continuously advertised none
  of them and matched nothing. The refusal even says...

v0.1.0 — Initial public release

First public release of the GitHub Actions runner fleet and dashboard.

Originally developed as private tooling for a macOS runner fleet; this release
generalises it for public use on Apple Silicon Macs.

What's included

• Runner registration and management — register.sh, status.sh, health.sh, cleanup.sh
• Preflight check — infers host requirements from cached workflow data
• Dashboard daemon — live fleet view, run history, drift detection, analytics, autoscaling
• Queue cause classification — diagnoses why jobs are queued (8 classified causes)
• Drain and resume — graceful runner shutdown with hook integration
• Admission control — job execution throttling via runner hooks
• Historical simulation — what-if analysis against real job history
• Burst forecasting — next-hour demand estimates with shadow evaluation
• Diagnostic bundles — allowlist-based redacted runner diagnostics
• Ephemeral runners — one-job clean builds
• Federation — coordinate runners across multiple Macs

Requirements

• macOS 14+ on Apple Silicon
• Homebrew at /opt/homebrew
• gh CLI authenticated
• Node >= 22.5.0 (for the dashboard)

Getting started

git clone https://github.com/addisdev/actions-runners.git ~/actions-runners
cd ~/actions-runners
./preflight.sh
./register.sh owner/your-repo
cd dashboard && ./fleetctl.sh install

See the Installation guide
for the full setup walkthrough.

Security

⚠️ Self-hosted runners on public repositories are not supported — only private repos with trusted contributors.

Read SECURITY.md and security-hardening.md before deploying.

Known limitations

• macOS on Apple Silicon only (no Intel, Linux, or Windows)
• Screenshots and demo mode not included in this release (see docs/images/README.md)
• autofix/escalate/ AI escalation subtree is disabled by default and has not been independently audited