Fix doctor telemetry aggregation: trust /health when telemetry DB missing, recover from backoff

[thenotoriousllama]

Why

Doctor relays the fleet health Hive renders. Two bugs made healthy services look broken: (1) when a service's telemetry SQLite DB was missing/unreadable but its /health probe was ok, doctor force-degraded it (→ honeycomb "degraded" whenever it wrote telemetry to a path doctor couldn't find); (2) a service that hit the restart give-up threshold stayed pinned unreachable even after it recovered — fatal for restartPolicy:"external" services (nectar) that doctor never restarts itself.

What

• ingestion/poll-loop: on a telemetry-DB fault with a healthy /health, report ok and still surface telemetryFault; a failing probe is unchanged.
• supervisor: reset backoff / consecutiveRestartFailures on an observed transition back to answering (including external-policy services), then remediate fresh from rung 1 if still degraded.

Tests

npm run ci green: 807 tests. Updated poll-loop tests; new supervisor FIX-B block.

🤖 Generated with Claude Code

Summary by CodeRabbit

• Bug Fixes
  • Improved service health reporting so a healthy probe can stay ok even if telemetry storage is missing, locked, or malformed.
  • Added clearer handling for degraded/unreachable services so known telemetry is preserved without exposing stale or invalid data.
  • Fixed supervisor recovery behavior so services can recover from a stuck backoff state when reachability returns, avoiding unnecessary remediation.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 34393135-b6f4-4f47-8604-07ca5f2d3879

📥 Commits

Reviewing files that changed from the base of the PR and between 68ca724 and 800652a.

📒 Files selected for processing (4)

• src/ingestion/poll-loop.ts
• src/supervisor.ts
• tests/ingestion/poll-loop.test.ts
• tests/supervisor.test.ts

---

📝 Walkthrough

Walkthrough

The poll loop's telemetry-fault handling now preserves coarse "ok" health when the /health probe succeeds, even with a missing, malformed, or mismatched telemetry DB, while still surfacing telemetryFault. The supervisor adds a recovery-reset branch when transitioning from unreachable to degraded, clearing stuck restart/backoff state. Tests updated accordingly.

Changes

Health probe masking and supervisor recovery reset

Layer / File(s)	Summary
Poll loop telemetry fault masking src/ingestion/poll-loop.ts, tests/ingestion/poll-loop.test.ts	An ok /health probe now keeps coarse health: "ok" despite telemetry DB faults (missing/malformed/mismatched), returning empty metrics/deeplake: null and telemetryFault; a new regression test confirms failing probes still report "unreachable".
Supervisor reachable-again recovery reset src/supervisor.ts, tests/supervisor.test.ts	tick() detects transitions from unreachable to degraded while stuck in advanced failure state, resets backoff/restart counters and rung to baseline, logs tick.reachable_again, and skips remediation that tick; new tests verify reset, subsequent normal remediation, and that degraded-to-degraded transitions don't force a reset.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Sequence Diagram(s)

sequenceDiagram
  participant PollLoop
  participant HealthProbe
  participant TelemetryDB
  PollLoop->>HealthProbe: check /health
  HealthProbe-->>PollLoop: probeHealthState
  PollLoop->>TelemetryDB: read telemetry data
  TelemetryDB-->>PollLoop: fault (missing/malformed/mismatch)
  alt probeHealthState is ok
    PollLoop->>PollLoop: keep health "ok", null metrics/deeplake, set telemetryFault
  else probe degraded/unreachable
    PollLoop->>PollLoop: health=probe state, fallback to last-known values
  end

Loading

sequenceDiagram
  participant Supervisor
  participant PersistedState
  participant RemediationRung
  Supervisor->>PersistedState: read lastKnownHealth, backoff, restart failures
  alt unreachable to degraded transition
    Supervisor->>PersistedState: reset counters, rung=1, lastKnownHealth=degraded
    Supervisor->>Supervisor: log tick.reachable_again, skip remediation
  else no recovery transition
    Supervisor->>RemediationRung: run normal remediation
  end

Loading

Possibly related PRs

• legioncodeinc/doctor#6: Introduces the pollEntry ingestion polling loop in src/ingestion/poll-loop.ts that this PR modifies for coarse health classification during telemetry faults.

Poem

A probe says "ok", the DB may lie,
but health stays true beneath the sky. 🐰
The supervisor wakes from its stuck despair,
resets its rungs with gentle care.
Hop, hop, recovered — tests all green,
the cleanest state this bunny's seen! 🥕

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarizes the two main changes: trusting /health when telemetry DB access fails and recovering supervisor backoff state.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/fleet-connectivity-live-reload

---

_{Comment @coderabbitai help to get the list of available commands.}