Agent → Watchdog supervision (Layer A4) — should it live in the agent, the watchdog, or somewhere else?

[bdunncompany]

Background

#852 closed the watchdog-restarts-agent direction (Layer B): watchdog catches stale heartbeat → restart BreezeAgent → verify recovery → escalate to FAILOVER on 5+ restarts/24h.

The spec for that work explicitly scoped out the inverse direction:

"Changing the heartbeat HTTP send path or anything in the main agent. The whole point of Layer B is that the main agent is unreliable; we fix this in the watchdog."
— docs/superpowers/specs/2026-05-22-watchdog-auto-restart-design.md:54

That makes sense for the wedged-main-agent case. But there's a sibling failure mode the watchdog can't catch by itself: the watchdog gets stopped while the main agent is healthy.

Failure modes #852 does NOT cover

Trigger	What happens today
Admin runs sc stop BreezeWatchdog during maintenance and forgets to start it	Watchdog gone, agent runs forever with no supervisor
A hung-update auto-disables the watchdog service	Same
Watchdog process crashes hard (segfault, panic, OOM kill)	#853 (SCM recovery actions, this morning) restarts it within 5s
Watchdog sc config gets set to Disabled by a misapplied GPO	#853 doesn't help — SCM won't restart a Disabled service. Agent has no way to re-arm it.

The "Disabled by GPO" + "manually stopped" cases are real for managed-endpoint fleets. The watchdog itself can't recover from being stopped — by definition.

Proposal (Layer A4)

A Windows-only goroutine inside startAgent (agent/cmd/breeze-agent/main.go) that:

• Every 60s, opens the BreezeWatchdog service via SCM
• If found in Stopped or Paused state → calls s.Start()
• Ignores *Pending transients
• After 3 consecutive open-or-start failures → backs off to 5× interval (slow-tick) so a deliberately-removed watchdog doesn't flood the agent log; still keeps trying so a reinstall is auto-detected
• Cancelled before shutdownAgent writes ShutdownIntent, so a clean stop doesn't race a supervisor-driven restart
• Linux/macOS: no-op stub (launchd/systemd already supervise breeze-watchdog at the OS level)

Conservative by design:

• Does NOT reinstall a missing service (bootstrap territory)
• Does NOT touch SCM config, only s.Start() on a stopped service
• Re-uses the existing seam-based test pattern from recovery_test.go (fake SCM controller, no real svc.mgr in tests)

The philosophical question

This puts watchdog supervision inside the main agent. The Layer B spec said "the main agent is unreliable; we fix this in the watchdog." Two ways to read that:

(a) Strict reading: nothing reliability-critical should live in the main agent, period. Then Layer A4 should live in a third process (or maybe in breeze-watchdog itself, monitoring its own re-arm path — though that's circular).

(b) Scoped reading: that constraint applies specifically to fixing main-agent wedges. Supervising a sibling service is a different concern — if main agent is healthy enough to talk to SCM, it's healthy enough to supervise the watchdog.

I read (b), and the code is written that way. But if Todd reads (a) and wants this in breeze-watchdog instead, that's a real difference. Posting here before sending a PR.

What's already written (local branch)

fix/agent-watchdog-supervision on my fork, one commit (fc3bf7de):

• agent/cmd/breeze-agent/main.go — registers supervisor goroutine in startAgent, cancels in shutdownAgent
• agent/cmd/breeze-agent/watchdog_supervisor_windows.go — Windows SCM logic
• agent/cmd/breeze-agent/watchdog_supervisor_other.go — Linux/macOS no-op
• agent/cmd/breeze-agent/watchdog_supervisor_windows_test.go — seam-based tests via fake SCM

Can rebase + file as a PR within 10 min if the direction is agreed. Happy to redirect it into breeze-watchdog if that's the call.

[ToddHebebrand]

@bdunncompany — spinning up a Project board for this so the close-out work + your A4 proposal land in one place: Watchdog Supervision (Discussion #854)

Seeded 14 tracks. Three things stand out as blocking before code:

1. Where A4 lives (your direct question). I'm leaning (b) scoped reading — agent goroutine for v1, with the explicit caveat that the agent-supervises-watchdog goroutine is narrowly scoped to "is the sibling service in Stopped state? if so, Start it." Not config repair, not restart-from-disabled, no growing scope into general process supervision. The Layer B "main agent is unreliable" constraint argues against putting reliability-critical logic in the agent in general; but A4 isn't reliability-critical for the watchdog — it's a best-effort self-heal, the watchdog still works without it. Worst case A4 fails silently and we fall back to the operator noticing.

I'd want to draw a hard line: A4 doesn't grow. If we ever need supervision of OTHER sidecars (helper, broker, future PAM broker, future ZTNA gateway), that's a separate design conversation — see the "Design doc: should A4 grow into a generic sidecar supervisor?" item on the board.

2. The GPO-disabled gap. Your discussion explicitly calls out the case neither A4 nor #853 closes — SCM refuses to start a Disabled service, and A4 only does s.Start(). Two options here, both on the board:

• Watchdog periodically re-arms its own SCM start type → bigger trust ask, watchdog modifying its own config
• Defer to a future installer-repair / bootstrap layer

I'd defer for now and document the gap. GPO-disable is an active misconfiguration; expecting silent self-repair invites surprises for the admin who did it on purpose. But if you have an opinion either way, say.

3. Linux/macOS A4. Your proposal makes A4 Windows-only since launchd/systemd already supervise. That's right for the stopped case, but neither covers launchctl disable / systemctl disable. For v1 I think the right call is document-the-gap, parity with Windows GPO-disabled handling. Same rationale.

If the answers above match your read, please rebase fix/agent-watchdog-supervision (commit fc3bf7de) onto current main and open the PR. Project board has the surrounding work (telemetry event, slow-tick backoff test, E2E on the Windows test VM) as separate items so you can ship A4 minimally first.

Also relevant on the board:

• PR fix(installer): apply SCM service-recovery actions on MSI deploys #853 (MSI sc failure recovery actions) — still open. Defense-in-depth with feat(watchdog): auto-restart on prolonged heartbeat silence (#799) #852: handles the crash case while feat(watchdog): auto-restart on prolonged heartbeat silence (#799) #852 handles the wedge case and A4 will handle the stopped-and-forgotten case.
• Web UI gap — mainAgentSilentSince and watchdogStatus are in the schema and event bus but no UI surfaces them yet. Amber "agent silent (watchdog OK)" badge and watchdog-status block on Device Detail are seeded as next steps.
• Auto-restart_main_agent after 5min silence — server-side recovery action you flagged in feat(asymmetry): server detects main-agent silence while watchdog stays online (#800 Layer C, server slice) #851's roadmap, default-off opt-in.

Direction approved on the discussion's core question. Schema/server pieces are unblocked; A4 implementation is unblocked pending your sign-off on the scope above.

[bdunncompany]

Direction agreed on all three scope constraints — A4 stays narrowly bounded (Stopped/Paused → Start(), no growth into general sidecar supervision, GPO-disabled deferred, Linux/macOS gap documented). Filed as PR #860, branched directly onto LanternOps thanks to the collaborator invite. Trivy + govulncheck + cross-platform build all clean per scripts/security/preflight.sh --fast.

[ToddHebebrand]

Closing — #860 landed 2026-05-26 implementing direction (b) from this discussion (agent-side BreezeWatchdog SCM supervisor with Stopped/Paused-only restart, seam-based testing via watchdogServiceController, no growth into generic supervision, Linux/macOS document-the-gap). All blockers from the original review were addressed in the fix-up commit; recovery cadence reset works correctly. Thanks for the thorough discussion — the scope constraints we agreed on here carried through to merge.