v0.3.0 — More than one Mac, one verdict, Fleet Cockpit

Full changelog: v0.2.0...v0.3.0 · Upgrading from an earlier version: one-time steps · Docs: addisdev.github.io/actions-runners

Thanks to @wized2 for the inferred-grouping tests (#22), the first contribution from outside the project.

More than one Mac, and one answer for the whole fleet. A second host joins in
one command, standby tiers run CI on a primary host first, and a switch takes a
host out of rotation. The fleet verdict reduces everything the dashboard knows
to one ordered answer, and Fleet Cockpit, its CLI and MCP server carry it to
the menu bar and to agent sessions. The dashboard also works away from the
desk: LAN and Tailscale access, per-device pairing, and a mobile layout. The
autofix bridge is removed.

Added

• Host on/off switch. A switch beside each host's name in the Fleet and
  Hosts tabs (on a single host, above the runner grid) takes the whole host
  out of CI: every runner drains with owner host, busy ones after their
  current job, and switching back on resumes only those. The confirm names the
  jobs still finishing and any repo no other host serves. host.off/host.on
  actions (agents too; cockpit run --host), scripts/host-drain.sh --off/--on/--status,
  host marker first line off. An off host is unusable to tiers and
  placement, refuses registration, and gets nothing from agent autonomy.
  /api/hosts gains off, drainingCount, activeCount and impact;
  /api/state gains hostImpact. The old host drain is now the secondary
  Pause new runners. See
  Taking a host out of rotation.

• Jobs orphaned by a draining runner are named, not waited on. A job that
  passed every step but whose runner went away during "Complete runner" stays
  in_progress on GitHub until someone force-cancels the run (addisdev/lake-api
  PR #73, 2026-10-12/13: three jobs, 10-25 minutes each, while cockpit why said
  "0 of 1 done" and cockpit wait kept waiting). The cockpit now asks GitHub
  for the jobs of runs active past three minutes; a job whose work steps have
  all completed, stuck at a "Complete …" step for three minutes, with its runner
  draining, offline, dead or lost (or with "Complete runner" cancelled, which
  only a runner shutdown does) is reported as orphaned by a draining runner —
  not your code, with the force-cancel and gh run rerun commands.
  cockpit why lists it; cockpit wait and wait_for_checks stop with exit
  2; cockpit wait --pr N --recover force-cancels and re-runs that PR's own
  orphaned runs once each (never a run whose work failed), logs each to
  ~/Library/Logs/FleetCockpit/orphan-recover.ndjson, and keeps waiting.
  CockpitCore/OrphanedJob.swift.

• A full shared memory table is now a verdict, not a mystery red test.
  hooks/shm-reap.sh logged orphan sweeps and low events to
  dashboard/logs/shm.ndjson, and nothing read it; the shmget error lives in
  step logs, which no failure class reads, so cockpit why called those
  failures ordinary. fleetd.js and agent.js now sample ipcs -m -o -p each
  tick (orphans the next hook will remove count as free) and summarise the log
  tail into the host vitals (host.shm, glance shm* vitals). A new shm-low
  rung, between dead-service and saturated, opens when fewer than
  FLEET_SHM_LOW_FREE segments are free on any reporting host or a low event
  is under 30 minutes old (critical when the table is full), with an alert of
  the same name. Host cards show free segments and orphans reaped in 24 h, the
  leak rate. The cockpit decodes the rung, the vitals and the history lane, and
  cockpit why flags it beside a repo's failures. dashboard/lib/shm.js.

• Standby tiers. Run CI on a primary host first and bring a standby
  host's runners online only when the primary is at its admission cap for 60 s,
  has a run queued over 2 min with no idle matching runner, or is unfit
  (disk-floor, saturated, a dead primary runner, critical memory pressure,
  or a stale heartbeat); drain them again after 10 min of headroom and an empty
  queue. A floor set of standby runners stays online always. While a primary
  lane is at cap its idle runners are drained too, so GitHub stops assigning
  it jobs it would only hold, but never without an online standby twin.
  dashboard/lib/tiers.js, wired into the fast tick, FLEET_TIERS_*
  (off/observe/enforce), shown as tiers on /api/state and the glance.
  See Standby tiers.

• drain-runner.sh --by=<owner> records who drained a runner, and
  --resume --by=<owner> only clears that owner's marker, so the controller
  never resumes an operator's drain. drain-stop-when-idle.sh keeps the owner.

• Agent: tiers.drain/tiers.resume batch commands, an optional
  FLEET_AGENT_COMMANDS allowlist (drain/resume only), and the autonomy rule:
  after FLEET_AGENT_AUTONOMY_S (180 s) without a heartbeat it resumes the
  runners the controller drained on its host.

• scripts/join-host.sh preflight also checks the Xcode license (xcodebuild -version answers without it, then swiftc exits 69 in the first job), an
  optional FLEET_XCODE_VERSION every host must match (a second host with an
  older Xcode builds the same job against a different SDK), and PyYAML for both
  python3 interpreters a job can reach. All three were hit adding a second
  host.

• Preflight checks plain CLI tools. preflight.sh checked Xcode, Postgres,
  Java, node and gh but nothing a Makefile calls, so a repo's make lint met
  make: shellcheck: No such file or directory on a new host (2026-10-09).
  scripts/cli-tools.txt lists the tools (shellcheck, jq, make always; deno
  and ruby when a self-hosted workflow runs them, via infer-checks.py), and
  scripts/check-tools.sh looks them up on the runners' PATH. join-host.sh
  checks every listed tool and --install-tools brew-installs the missing
  ones. Preflight also warns when the runners' python3 is PEP 668 externally
  managed. Tests: scripts/test-preflight-tools.sh.

• Admission decisions from every host. An agent host's hooks log locally
  like the coordinator's, and nothing read that file, so holds on a second host
  never reached the dashboard or the admission-hold alert. The agent now ships
  the lines added since its last accepted heartbeat (lib/admission-ship.js,
  at-least-once, offset advanced only by the coordinator's count), and the
  coordinator stores them by host: admission_events.host_id, waiting[].host
  and admission.byHost. The headline mode and limit stay the coordinator's
  own. host_samples.host_id records which machine a vitals row describes, so
  moving the coordinator does not splice two machines' load into one history.
  Runbook: docs/federation.md, "Moving the coordinator to another Mac".

• One-command second host. scripts/join-host.sh joins a Mac to the fleet:
  preflight (including the toolchains jobs use), the agent, mirrored runners and
  the health timer, dry run unless --apply. scripts/mirror-runners.sh and
  dashboard/lib/mirror.js copy the coordinator's runners one per distinct
  label set, skipping what the host cannot run (a missing label, Simulator
  runners without Xcode, FLEET_MIRROR_SKIP_REPOS), with registration tokens
  optionally minted elsewhere (--tokens-from). Template:
  examples/fleet.env.second-host. The agent and installer accept the token
  from FLEET_AGENT_TOKEN_FILE. Runner pin 2.336.0 → 2.337.0.

• Fleet verdict. lib/verdict.js reduces drift, queue causes, failure
  classes, admission holds and alerts to one ordered ladder — cannot read
  GitHub, host down, disk floor, dead service, saturated, account blocked,
  config drift, waiting, clear — with evidence and one next move, plus a state
  for every runner. Published as verdict on /api/state. Checked against
  recorded incidents in test/fixtures/scenarios.js.

• GET /api/glance and /api/stream?view=glance: a versioned
  (schema: 1) ~10 KB payload for small screens and slow links, where the full
  state is ~130 KB.

• admission-hold alert. A job held by the admission disk floor opens a
  critical alert after 2 minutes — previously jobs sat at "Set up runner" with
  no error anywhere. A slot wait past FLEET_ADMIT_MAX_WAIT_S opens a warning.

• host-saturated alert. Two runner-lost jobs inside an hour — the
  signature of a starved host — open a warning while it is happening.

• scripts/scrub-snapshot.mjs turns a live snapshot into a committable fixture.

• Fleet Cockpit (cockpit/), a macOS menu bar client of /api/glance:
  verdict, per-host vitals with the disk floor drawn on the gauge, a pill per
  runner, the queue. Reaches a loopback-only dashboard through an SSH tunnel
  that dies with the app; reconnects on wake and network change; keeps a stale
  view greyed rather than green. Bundled incident fixtures, --render for
  screenshots, a cockpit CLI, and a snapshot file for other tools. See
  docs/cockpit/.

• scripts/make-glance-fixtures.mjs keeps the cockpit's fixtures in step with
  the daemon (checked in CI).

• Out-of-band sentinel in the cockpit: when the dashboard does not answer it
  reads SSH reachability, GitHub's view of the host's runners, a runner on
  another machine and githubstatus.com against one table — host asleep, power
  or network out, rebooted with nobody logged in, every service dead, dashboard
  down with the fleet working, this Mac offline — and notifies on transitions.
  cockpit sentinel runs the same check from a terminal.

• The cockpit's Why? ladder, Copy brief (markdown incident brief, also
  cockpit brief), and Failed in the last 2 hours with a not your code
  mark. /api/glance gains failures.

• Queue ETAs (lib/eta.js): each queued run's start and finish as
  p50–p90 ranges from the job ahead of it and the workflow's history; none for
  causes that never clear. /api/glance gains runs, recent and the ETA
  fields.

• Checks rollup and watches in the cockpit: one row per commit with
  progress and time to green; a bell to be notified when it finishes.

• Agent CLI: cockpit why <repo>, cockpit queue, and cockpit wait (exit
  2 the moment waiting is pointless), plus a Claude Code skill in
  cockpit/skills/fleet-cockpit.

• Cockpit control. Pair a Mac (device token in the Keychain, minted with
  fleetctl.sh pair over SSH); buttons from the action catalogue with its
  confirmation text; a runner panel with recent jobs, events and restart /
  drain / resume. Alert notifications on transitions with Repair, Dismiss
  everywhere and Snooze actions, a storm summary, critical-only mode, quiet
  hours and muted rules. fleetcockpit:// URLs (pair, why, runner, reconnect,
  fixture). CLI: cockpit pair, cockpit run <action> (--yes for anything
  that changes the fleet).

• Disk floor forecast (lib/disk-forecast.js): 6 h and 72 h fits against
  the admission floor, backtested on the 2026-09-28 freeze (warned 6+ hours
  ahead); a disk-floor-soon alert under 12 hours.

• GET /api/posture (lib/posture.js): standing risks with the fix and who
  can apply it. GET /api/timeline (lib/timeline.js): incident intervals
  per ladder rung, today's queue vs build time, the week, flaky runners,
  sparkline samples.

• Cockpit: History panel, sparklines, standing-risks list, weekly digest,
  "What is using disk?", top CPU with a Spotlight detector (cockpit top),
  diagnostic bundle download.

• Verdict banner on the web dashboard's Fleet and Runs tabs.

• Cockpit: desktop widgets (app group snapshot), App Intents (Fleet Status, Why
  Is a Repo Queued, Repair Fleet) with Siri/Spotlight phrases, a Focus filter,
  a ⌃⌥⌘F floating panel, an incident replay window, and a sound on green.

• Host sentinel (scripts/host-sentinel.sh): from another machine, an ntfy
  message when the runner host stops answering and when it returns.

• cockpit mcp: an MCP server with fleet_status, why_queued,
  fleet_queue and wait_for_checks.

• Release workflow for signed, notarized cockpit builds, gated on secrets.

• LAN access. ./fleetctl.sh remote lan on|off sets FLEET_HOST in
  fleet.env and regenerates the LaunchAgent, which is where the bind address
  actually lives. ./fleetctl.sh remote status shows the running daemon's real
  bind, every reachable URL, the Serve config, and the firewall state.

• Tailscale Serve integration. ./fleetctl.sh remote tailscale on|off
  configures Tailscale Serve for HTTPS access on your tailnet, finding the CLI
  in Homebrew or the Mac App Store app bundle. off removes only the
  dashboard's listener. Never Funnel — the dashboard is never exposed to the
  public internet.

• DNS-rebinding protection. Browser requests are validated against the
  server's own identity (hostname, Bonjour name, LAN IPs, Tailscale MagicDNS,
  coordinator URLs, FLEET_ALLOWED_HOSTS). Unknown Host headers get 421
  before any route logic runs, and the refused name is logged. Agent routes are
  exempt; they are token-authenticated.

• Per-device pairing. ./fleetctl.sh pair generates a 6-digit pairing code
  (+ terminal QR if qrencode is installed) and a link on an address the phone
  can reach. It warns if no such address exists. Remote devices scan, tap, or
  type the code to get their own revocable bearer token; the master token never
  leaves the host. Guessing is capped per client and globally. The Control tab
  shows a QR code with a countdown, notices when pairing succeeds, and detects a
  revoked token. ./fleetctl.sh devices and ./fleetctl.sh revoke <name>
  manage devices from the terminal.

• Mobile bottom nav. Below 640 px: fixed bottom bar (Fleet, Runs, Alerts,
  Hosts, More), More sheet for secondary tabs, collapsing top bar.

• Mobile glance card. Busy / Idle / Queued / Alerts at the top of Fleet,
  counted the same way as the KPI row, each tappable to its tab.

• Hash routing. URLs like #/runs, #/alerts, #/hosts are bookmarkable,
  restorable on refresh, and shareable. The Back gesture closes an open drawer
  before it leaves the tab.

• Resilient SSE. Jittered exponential backoff on reconnect (1s → 30s cap).
  On return to a suspended tab it reconnects only if the stream has gone quiet.
  The last snapshot stays on screen through an outage. Elapsed timers pause
  while the tab is hidden.

• PWA improvements. viewport-fit=cover + safe-area padding for notches.
  apple-touch-icon PNG (180 px). Manifest shortcuts (Runs, Alerts). Service
  worker (sw.js) registered only on secure contexts (localhost / Tailscale
  HTTPS). It fetches the shell network-first, so upgrades reach phones
  immediately, and keeps the last /api/state for offline launch.

• Touch-friendly controls. Dismiss buttons always visible and 44 px targets
  under @media (hover: none). tip.js shows title text on tap, which touch
  browsers otherwise never display.

• Scrolling tables. Every table is wrapped in a table-scroll container
  with edge shadows, so wide tables scroll inside their panel on a phone.

• Connection pill states. "Offline", "Reconnecting…" (stream down, last
  data shown), "Stale" / "Collector stalled" (health check), "Live".

• Access banner. Non-local viewers see their connection context (LAN /
  Tailscale user) and a link to the pairing flow.

• Alerts tab now polls every 30 s while open, matching Hosts.

• /api/access endpoint returns viewer context (via, tailscale user, URLs).

• lib/remote.js — host allowlist, proxy-aware origin/sameOrigin, reachable
  URL list. lib/devices.js — pairing codes, hashed device tokens, rate limiting.

• dashboard/public/tip.js — tap-to-reveal popovers for touch devices.

• dashboard/public/sw.js — service worker for offline/PWA support.

• dashboard/public/vendor/qrcodegen.js — vendored Nayuki QR Code generator
  (MIT licensed, no npm dependency, no build step).

• New docs: docs/remote-access.md covering LAN, Tailscale, pairing, and
  the security model. mkdocs.yml nav updated.

Changed

• Hosts tile replaces the federation strip. The one-line strip above the
  summary tiles ("2 hosts · 110 runners online · 0 building · fleet headroom")
  crowded the row below it and repeated two tiles. A Hosts tile now leads
  the row when federation or the control plane is on: host count, stale hosts
  and fleet headroom, red when a host is stale, and a click opens the Hosts
  tab. The control-plane role and replica moved to the header line. Tiles
  narrow to a 132px minimum so all nine still fit one row at 1440px. The docs
  fixture fleet is now two hosts, so fleet-tab.png shows the tile.
• Automated agents stay off the operator's Waiting Board. register.sh
  writes WB_SKIP=1 into each new runner's .env, so CI jobs, which load the
  same user-level Claude Code and Cursor hooks as interactive sessions, never
  file waits or inbox items. Existing runners need the line added once;
  install-hooks.sh preserves it.
• The saturated verdict no longer fires on a run the headroom gate is
  holding; on a busy host that is most of the day, and it is waiting.
• Offline alerts: one per host outage, one per flap episode. Three or more
  runners offline in the same tick (offlineHostThreshold) now open a single
  offline-host critical, "N of M runners offline — host unreachable or
  saturated", and every offline runner folds into it while it is open. One or
  two runners still alert on their own. Offline alerts stay open for
  offlineDebounceMs (15 min) after the runner reconnects and close dated at
  the reconnect, so a runner that flaps is one alert. Replayed over the
  recorded intervals, 2026-09-20 goes from 1,284 offline alerts (264
  notifications) to 21 (41).
• lib/auth.js authorize() now accepts a checkDevice parameter so it can
  validate device tokens alongside the master token.
• isLocalRequest() and sameOrigin() moved from lib/auth.js to
  lib/remote.js. They are re-imported in fleetd.js — no external API change.
• isLocalRequest() now returns false when a loopback request carries proxy
  headers (Tailscale Serve). Previously, Tailscale requests looked local.

Removed

• The autofix bridge. dashboard/autofix/ is gone: the fleet-autofix
  LaunchAgent, infra reruns, Cursor-agent escalations and fix PRs, and with them
  the project's only Cursor SDK dependency. Alerts notify and nothing acts on
  them unattended; health.sh --repair remains the response to launchd-dead,
  launchd-missing and offline. GET /api/remediation-candidates,
  GET /api/autofix/status, FLEET_AUTOFIX_STATUS_URL and the dashboard's
  Remediation panel went with it. To clean up a host that ran it: unload and
  delete ~/Library/LaunchAgents/<prefix>.fleet-autofix.plist, and delete
  ~/.cursor/sdk/auth.json or dashboard/.cursor-api-key.

Fixed

• Primary self-drain flapped. The standby tiers controller drained every
  idle primary runner on the first tick a lane reached its cap and resumed them
  all 30 s after it dipped below. With a build cap of 2 that is the lane's
  normal operating point: from 10-10 to 10-13 runner-host saw 489 drain batches
  (up to 55 runners each, 7,495 runner drains), a median hold of 2.9 min, a
  resume followed by the next drain within 2 min half the time, and 88 drains
  that landed on a runner which had just taken a job. Self-drain now has
  hysteresis per lane: it starts after 60 s at cap
  (FLEET_TIERS_SELF_DRAIN_AFTER_S), ends after 5 min with room
  (FLEET_TIERS_SELF_RESUME_AFTER_S, was 30 s) and no sooner than 10 min after
  it started (FLEET_TIERS_SELF_MIN_HOLD_S), and does not start again within
  5 min of ending (FLEET_TIERS_SELF_MIN_OPEN_S). Resumes that keep a repo
  served are still immediate, and a lane in its hold never drains more runners
  unless it is at cap. /api/state lanes gain shedding and shedSince.

• Drained runners came back after a reboot or login, because their
  LaunchAgents run at load and svc.sh stop only unloads. fleetd and the agent
  now stop a running drained runner again within a tick, keeping its drain
  owner (FLEET_HOLD_DRAINS, default on).

• Draining a busy runner orphaned its job. drain-stop-when-idle.sh is
  started by the completion hook, which runs under Runner.Worker, and macOS
  pgrep leaves the caller's ancestors out of its match without -a. So the
  script never saw the worker it was waiting for and stopped the service at
  once, mid-hook; the listener cancelled the job before it could report and
  GitHub kept it in_progress. runner-host's worker logs show 40 jobs on 15+
  runners shut down inside the hook from 10-10 to 10-13, as the tiers controller
  drained runners that had just taken a job. Both drain scripts now use
  pgrep -a, and the stop ignores the SIGTERM that unloading the runner's
  launchd job sends its process group, which used to kill the script before it
  could mark the drain done (the marker stayed draining).

• Killed Postgres clusters exhausted the host's shared memory table. A
  cancelled or timed-out job kills its throwaway cluster without a clean
  shutdown, and the cluster's System V segment stays. macOS allows 32, so after
  about thirty such jobs every Postgres suite on the host failed with
  could not create shared memory segment: No space left on device
  (2026-10-13: 31 orphans removed by hand; four more within half an hour). Both
  job hooks now run hooks/shm-reap.sh, which removes only segments owned by
  the runner user with NATTCH 0 and a dead creator PID, and logs to
  dashboard/logs/shm.ndjson. FLEET_SHM_*.

• cockpit wait read a failed attempt under its in-progress re-run. It
  (and the MCP wait_for_checks) returned red at once while a re-run of the
  failed run was in progress (jot-ios PR #181, run 38163415231,
  2026-10-11: attempt 2 failed, gh run rerun --failed started attempt 3, and
  the wait printed "1 failed · 1 of 1 done" twice). A re-run keeps the run id,
  and the daemon's view still held attempt 2. Before a red ends the wait, each
  failed run is now asked of GitHub for its latest attempt; one running again is
  pending, and that row is laid over cockpit's view until the view catches up.
  The daemon merges its run pages by latest attempt (it kept whichever copy
  arrived first) and publishes runAttempt on glance runs; the rollup keeps one
  row per run id, its latest attempt.

• A runner that had updated itself read as idle mid-job. Updated runners
  start Runner.Worker from a versioned bin.<version>/ directory, and the
  daemon, drain-runner.sh, drain-stop-when-idle.sh, deregister.sh and
  install-hooks.sh only looked under bin/. On the coordinator every busy
  runner showed idle (2026-10-09), and draining one stopped its service under
  the running job instead of waiting for it. All five now match both.

• cockpit wait could decide from GitHub before the first glance landed.
  The loop treated "no glance yet" as a missing fleet view from the wait
  start and accelerated the GitHub cross-check while the SSE connection was
  still opening; on a loaded runner GitHub could answer green before cockpit's
  frozen progress line and qualified repo slug were set. Grace for the first
  glance is anchored to when the stream opens; the fast GitHub cadence applies
  only to a stalled collector or a dashboard with no open stream.

• cockpit wait outlived its deadline by hours. On 2026-10-09 three
  cockpit wait --pr sessions (quant-api #54, quant-web #116,
  map-web #11) ran 1 h 37 min with no timeout while all three PRs were
  merged and green. sample put each one in -[NSConcreteTask waitUntilExit]
  inside the GitHub cross-check, with the gh child already gone: gh had
  overrun its 20 s budget, the timer called terminate() while a worker sat
  in waitUntilExit(), and Foundation never marked the task finished (a
  stand-alone repro hangs within a few tries). The loop awaited that ask
  inline, so neither the deadline nor cockpit's own (live) stream could end
  the wait, and the wait's SSH tunnel stayed up with it. Now Subprocess runs
  every short-lived program without waitUntilExit and always answers by its
  deadline; the wait's deadline is a wall-clock timer of its own
  (Deadline.race), each GitHub ask has its own budget, and the CLI exits 3
  thirty seconds past --timeout whatever else is happening. A stream silent
  for 75 s is torn down and its tunnel reopened, and a view with no runs for
  the PR asks GitHub at once instead of after two minutes (short --fresh
  retries had exited "no runs seen" on a green PR).

• The Fleet tab's runner grid rendered nothing on a multi-host fleet.
  projectGroups() passed a snapshot it never received to runnerTile(), so
  every federated render threw s is not defined before the host blocks
  mounted. It now takes the snapshot.

• Runners ran jobs with whatever PATH the shell that registered them had:
  config.sh copies the caller's $PATH into .path, and the runner gives
  jobs that, not .env's. runner-host had six PATHs (17 runners with no
  Homebrew, nine with nvm's Node and SnowSQL from an interactive shell), and a
  second host's python3 resolved to Homebrew's 3.14 where its twin's was
  Xcode's 3.9, which broke kit-ci there. register.sh writes
  FLEET_RUNNER_PATH into .path after config.sh; scripts/runner-path.sh
  aligns existing runners (idle ones only, dry run by default) and join-host's
  preflight reports drift.
  runner-path.sh --apply leaves a drained runner (.drain) stopped: it
  writes the new PATH but no longer starts it, which had undone every drain.

• Lint and queue-cause false positives. The Lint tab reported an archived
  repo CRITICAL ("no runner is registered") because the roster skipped archived
  repos without recording them, so their cached workflow files lived on; files of
  deleted repos (workflow list 404) were also kept forever. Both are now dropped,
  and the lint skips archived and unknown repos. hosted-macos no longer fires
  on public repos (the upsert never refreshed private, so a repo made public
  stayed private in the table) and is info when the job's if: reads the repo's
  visibility. no-cancel-in-progress accepts an expression that reads the event
  or the ref (any other expression is info) and counts job-level concurrency;
  cached workflow files of repos gone from the roster are dropped even when the
  old name still redirects, and the upsert refreshes name too. Label findings on a
  job gated by if: vars.* are info. The queue-cause classifier matched lint
  findings per repo, so a queued CI run behind a busy runner was called
  label-mismatch because a different workflow had a finding; it now matches
  the finding's workflow file, branch and job labels (lintFindingForRun).

• An agent configured with FLEET_COORDINATOR alone reported to nobody:
  agentctl.sh writes both coordinator keys into the plist, the unused one
  empty, and agent.js read them with ??, which takes an empty string as set.
  Found by the first real second host.

• health.sh no longer goes red when GitHub does not answer. A failed
  runner API call was read as "this runner is unhealthy", so a rate-limit or
  network blip turned the health job red for every runner at once (red in 147
  of 2617 runs on runner-host; every recent one was such a blip). It now says GitHub did not answer and still judges
  launchd; a runner missing from a real answer is reported not-registered
  and stays a fault. Covered by scripts/test-health.sh.

• cockpit wait sat on a frozen view and never returned. On 2026-10-07
  runner-host's fast loop finished its last tick about 12:32Z under load.
  fleetd publishes only after a tick, so every client kept that glance
  (shop-ios PR #445 "0 of 1 done", shop-android PR #280
  "instrumentation · 12 min") while keepalives held the stream open, and both
  checks went green on GitHub minutes later. The wait decided, and checked its
  deadline, only when a glance arrived, so it could neither finish nor time
  out. WaitLoop (CLI and MCP) runs on a timer. With --pr it also asks
  GitHub itself (gh pr view --json statusCheckRollup): every 30 s while
  cockpit's view is over 4 minutes old or the dashboard is unreachable, every
  2 minutes otherwise. GitHub's finished answer ends the wait and is labelled
  as GitHub's. An unreachable dashboard no longer ends a PR wait with exit 3.
  A view whose last tick is over 4 minutes old now reads "Collector stalled" in
  the app, status and why, instead of the old verdict. why also shows
  the same verdict as status now; it had printed the daemon's verdict even
  when the app knew the host was down.

• The disk floor held the fleet with tens of GB to spare. df counts macOS's
  purgeable caches as used, and macOS purges only under real pressure, which a
  held, idle fleet never creates: on runner-host df said 101 GB free while
  macOS offered 163 GB (2026-10-07). The admission floor, cleanup.sh and the
  dashboard now measure usable space (plain free plus purgeable,
  hooks/free-disk.sh) with a 15 GB hard floor on plain free
  (FLEET_ADMIT_MIN_PLAIN_FREE_GB). The hook tests now run the hook under
  bash -e as the runner does. That exposed two ways a failed disk read could
  exit the hook, whose EXIT trap then admitted the job without a slot; both are
  closed.

• runner-host kept reaching the disk floor. The weekly cleanup reclaimed
  1–2 GB, refused to run while any job was building, and never touched the
  largest growing thing: each runner's ci- simulator, 48 → 62 GB across 41
  devices in three days. cleanup.sh --auto now runs every 15 minutes and
  acts under a pressure line (floor + 20 GB) or once a day. It erases idle CI
  simulators over 3 GB (1 GB under pressure), under pressure removes idle
  runners' Playwright browsers, and skips only the runners that are building.

• The fast loop stopped for good when one tick never finished. A tick
  overran at 2026-10-06 16:31Z and every tick after it was skipped as an
  "overlap" (235 on 10-06, 1,034 on 10-07), while health read ok because
  the slow loop also runs the tick. lib/tick-guard.js now waits a stuck
  tick out for FLEET_FAST_ABANDON_MS (4 min), then abandons it and starts
  the next. At most FLEET_FAST_MAX_ABANDONED (3) may be outstanding;
  past that health goes not-ok. Every overrun, skip and abandon names the
  stage the tick was in, and /api/health reports fastLoop.

• A cancelled job held by admission kept its runner for minutes. The
  worker never signals the job-started hook, and GitHub keeps the run and the
  job in_progress until it force-completes the job, about four minutes later
  (measured 2026-10-07, map-web run 37236945666), so the run-status poll
  could not fire sooner. The hook now reads its worker's _diag log every
  poll and leaves as soon as the worker logs the cancel.

• cockpit wait --pr read red from a commit that was no longer the PR. It
  picked whichever of the PR's commits had the latest update, and nothing said
  which commit was the head. On site-web PR #79 (2026-10-03) the branch
  was force-pushed back off a failing commit, and before the daemon saw the new
  run, the wait answered "red" from the old commit. A wait on the re-pushed sha
  answered "cancelled" from that sha's first run. The glance's runs now carry
  prHead, the PR's head as GitHub reported it on that tick (from
  pull_requests[].head.sha, no extra API call). A PR's checks are the head's
  checks, and a head with no run yet is still waiting. A head whose result is
  older than a run of the same workflow on another of the PR's commits was left
  and re-pushed, so that result waits for the new run. Within a commit, the
  newest run of a workflow is now chosen by run id, not updatedAt: a run that
  concurrency cancelled after its replacement was created used to win.

• Held jobs never noticed their run had ended. Runner LaunchAgents set
  SessionCreate, so gh inside a job has no keychain token; the hook's
  cancellation poll went out anonymously and never saw completed. No held
  job had ever logged cancelled, and waits for runs that ended hours earlier
  stayed in the queue. The hook now asks the daemon first
  (GET /api/run-status, cached 20 s, fleet repos only), then gh, then
  curl. FLEET_ADMIT_STATUS_URL= (empty) skips the daemon.

• cleanup.sh missed superseded runner versions. A runner self-update
  leaves the previous bin.<version> and externals.<version> in place. At
  59 runners that was 25 GB, and it held the 40 GB disk floor shut for most of
  2026-10-05. Cleanup now removes pairs older than the linked version and
  leaves a newer, unlinked pair (an update in progress) alone.

• Admission stalled with every slot free. Every held job polled through the
  admission mutex, and each holder spent seconds reading the whole waiter queue
  on a loaded host, so the lock was never free and the oldest waiter, the only
  one that can be admitted, almost never won it. runner-host admitted nothing
  for a day with 40 jobs held, busy=0 and an empty reason (2026-10-03/04). A
  waiter with a live, non-Simulator waiter ahead of it now skips the mutex.

• Job rows froze at their last in-progress snapshot. The fast loop fetches
  jobs only for active runs and the backfill only for runs with no job rows, so
  a run's final jobs never got a conclusion, completed_at or duration once
  the run completed — about a third of the history (5,966 rows on runner-host),
  which made every build-hours figure undercount. lib/settle.js re-fetches
  each such job by id: on the tick its run is first seen completed, and in a
  new backfill phase for anything missed. The job upsert also now fills
  started_at and runner_id for a job first seen queued.
  scripts/settle-stale-jobs.mjs clears existing history in one go.

• Runs froze too. A run that finished after it dropped off GitHub's active
  lists and its repo's newest page kept its last active status (23 on
  runner-host, back to 2026-08-06), so its jobs never qualified for settling.
  The fast tick now refreshes up to four runs a tick that are active in the
  database but unseen for 10 minutes (RunSettler), then settles their jobs.

• settle-stale-jobs.mjs read its budget from gh api rate_limit, which kept
  answering 5000/5000 while four workers spent the per-user quota and got the
  daemon's token penalised for ~45 minutes. It now reads each response's
  rate-limit headers, defaults to one worker and a 2500 floor, and waits out a
  primary-limit error instead of exiting.

• A public repo queued for GitHub-hosted runners read as configuration drift
  ("will never be picked up") and made cockpit wait give up; it is ordinary
  waiting on GitHub. Private repos asking for hosted runners still count.

• The verdict counted jobs held by the admission hook as running ("8 running"
  on a host admitting 2); only busy runners count now.

• A standing-risk check that could not run (Spotlight's mdfind, which takes
  ~14 s over the fleet root and timed out under load) was dropped from the
  glance, which read as fixed. The probe now has a minute, and unchecked items
  are carried and shown as "unchecked".

• Workflows on hosted macOS in public repos (free) no longer count as a
  billing risk.

• Disk-floor deadlock. cleanup.sh counted runners held by the admission
  hook as busy, so during a disk-floor freeze the held jobs stopped the one
  script that frees disk (seen live 2026-09-30). Held runners are no longer
  counted, and the dry run always runs.

• A job held while disk was under the floor kept reading as disk-held after the
  disk was freed (the hold's reason is recorded once); the verdict and the
  admission-hold alert now count a disk reason only while disk is still under
  the floor.

• The coordinator was listed twice in hosts when a heartbeat under its own id
  had been recorded (an agent.js run on the coordinator); once that record
  aged out it read as a second host that had stopped reporting.

• dashboard/.fleet-vapid.json, the web-push private key, was not ignored.