Health Monitor

Health monitor

05-dns-health-monitor.rsc creates one script and one scheduler entry. It runs every interval (60 s default; 3 m is fine for a home device), keeps its state in globals and logs only on transitions, with an UPPERCASE tag (HEALTHY, SUSPECT, FALLBACK, …) so a log monitor can match them.

What it measures (two independent things)

1. Are the answers usable? Three canaries are resolved through the live resolver and judged by content: empty, 0.0.0.0 and the stub pair 8.6.112.0/8.47.69.0 count as unusable.
2. Does each endpoint answer? The primary (and, before a switch, the fallback) is probed with /tool fetch against its DNS-JSON API ("Status":0 in the body). This does not touch /ip dns.

Before either: device-mode and the clock. On a home device fetch and the scheduler are rejected, so the monitor logs MONITOR_DISABLED_BY_DEVICE_MODE and enters manual-only. A wrong date makes verify-doh-cert impossible; the script logs CLOCK_INVALID and stops, instead of reporting "DoH down" forever.

State machine

HEALTHY --canary fails--> SUSPECT
SUSPECT --canary passes--> HEALTHY
SUSPECT --failureThreshold bad runs--> FALLBACK   (fallback probed before, verified after)
FALLBACK --primary passes--> RECOVERING
RECOVERING --recoveryThreshold good runs--> HEALTHY
RECOVERING --primary fails again--> FALLBACK
SUSPECT / FALLBACK --both endpoints unreachable--> DEGRADED
any --answers are the stub pair--> STUB          (not a failover-able fault)
any --device-mode=home--> MANUAL-ONLY

State	Condition	Action
healthy	at least one canary returns a real address	primary active; reset counters
suspect	a bad run seen, below failureThreshold	keep the primary, keep counting
stub	answers are the stub pair	nothing — a second endpoint returns the same pair
fallback	primary failed failureThreshold runs, fallback verified	on the fallback endpoint
recovering	primary answers again, below recoveryThreshold	still on the fallback, counting up
degraded	both endpoints unreachable	keep the current configuration
clock-invalid	clock date wrong	stop — verify-doh-cert cannot succeed
manual-only	device-mode=home	no fetch/scheduler — run by hand, no failover

Anti-flap. failureThreshold (2) consecutive bad runs to leave the primary, recoveryThreshold (2) consecutive good runs to come home. With interval=60s a switch needs ~2 minutes in either direction, so a single blip does not move the router.

Vendor coupling. The fallback is a different operator (Google by default), so a failover changes the upstream and therefore its answers, filtering and geo behaviour. This is managed failover, not transparent failover.

Failover is verified and reversible

1. The fallback is probed before the switch — a broken fallback never takes the LAN down.
2. The switch is applied, then verified by resolving example.com.
3. If it does not resolve, the primary is restored and the log names the likely cause (missing CA for the fallback host).
4. At most maxAttempts (3) attempts per outage; the router is never left parked on a dead endpoint.

Hardening in 0.3.1

• Endpoint-driven failover. The content check resolves canaries through the live resolver, which serves from cache. A primary that dies while the cache still holds answers looked "healthy" and the failover was delayed by the cache TTL. Now a failover is triggered when the endpoint probe (probeA) is unreachable or no usable answers come back, so the switch happens before the cache expires.
• Concurrency guard. If a run is still active when the scheduler fires again, the new run logs DNS monitor: skipped — a previous run is still active and does nothing. The flag is cleared in an on-error handler, so a crashed run cannot leave the monitor locked.
• Recovery is gated on the primary. The monitor returns to the primary only when probeA == "ok"; otherwise it stays on the fallback (no A↔B flapping).
• Clock range. The monitor refuses to act on a date outside 2025..2035 (past and future break verify-doh-cert).

Hardening in 0.4.2

Two field findings changed how a verdict is reached.

1. A perfect-looking answer can be a substitution. A network that intercepts DNS returns Status: 0 with the stub address, so the JSON status proves nothing. The probe now looks for the configured stub prefixes inside the payload and reports substituted.

2. A stub can be a cache leftover. An answer cached while the path was intercepted stays until TTL expires, however good the resolver is now — measured: after pointing the router at an honest resolver, clients still received stubs until the cache was flushed. STUB is therefore declared only after one flush and a re-check; if the answers turn sane, the state goes to healthy with STUB_CLEARED in the log.

3. Own endpoints are probed differently. probeMode "resolve" judges the primary by answer quality instead of a DNS-JSON request. Use it whenever dohUrlA is your own resolver: otherwise a healthy primary is reported bad-status and the router is moved onto a public fallback, where answers are substituted.

Hardening in 0.4.0

• In-memory probe. /tool fetch ... as-value output=user returns the JSON body in memory, so there are no temporary files in /file and no cleanup races. The JSON check accepts "Status":0 and "Status": 0.
• Ordered chain. primary → secondary → tertiary (dohUrlC, optional; empty disables it) → plain DNS (plainFallback, optional, off by default). The plain step logs PRIVACY DEGRADED.
• plain state. On plain DNS; recovery to the primary when it answers again.

Test it in five minutes

# 1. simulate a dead primary
/ip dns set use-doh-server="https://192.0.2.1/dns-query"
/ip dns cache flush
:global DNSMonitorState; :set DNSMonitorState ""
:global DNSFailCount; :set DNSFailCount 0
:global DNSRecoverCount; :set DNSRecoverCount 0
:global DNSFailoverAttempts; :set DNSFailoverAttempts 0
:global DNSMonitorRunning; :set DNSMonitorRunning false
/system script run DNS_HealthMonitor_Failover    # run once -> SUSPECT (1/2)
/system script run DNS_HealthMonitor_Failover    # run again -> FALLBACK
/ip dns print                      # expect: use-doh-server = your fallback
/log print where message~"SUSPECT" or message~"FALLBACK"

# 2. restore (run twice to pass recoveryThreshold)
/ip dns set use-doh-server="https://cloudflare-dns.com/dns-query" verify-doh-cert=yes
/ip dns cache flush
:global DNSMonitorState; :set DNSMonitorState ""
/system script run DNS_HealthMonitor_Failover    # -> RECOVERING (1/2)
/system script run DNS_HealthMonitor_Failover    # -> HEALTHY

Other cases: both endpoints dead → DEGRADED: both endpoints unreachable; wrong year in /system clock → CLOCK_INVALID; device-mode=home → MONITOR_DISABLED_BY_DEVICE_MODE.

Log lines you will actually see

HEALTHY: answers look sane (3 of 3)
SUSPECT: primary unreachable (1/2)
SUSPECT: primary unreachable (2/2, attempts 0/3)
FALLBACK: switched to fallback DoH (https://dns.google/dns-query), example.com -> 104.20.23.154
RECOVERING: primary answers again (1/2)
HEALTHY: primary DoH answers again — switched back from the fallback
STUB: answers are the stub pair — example.com=8.47.69.0 (stub pair); …
DEGRADED: both endpoints unreachable (primary unreachable, fallback unreachable) — keeping the current configuration
DEGRADED: fallback DoH did not resolve either — rolled back to the primary; check the CA for the fallback host
CLOCK_INVALID: clock looks wrong (2026-10-26) — certificate validation cannot succeed; enable NTP
MONITOR_DISABLED_BY_DEVICE_MODE: device-mode=home — /tool fetch and the scheduler are unavailable; run the monitor by hand and expect no failover

Manual run and state reset

/system script run DNS_HealthMonitor_Failover
:global DNSMonitorState; :set DNSMonitorState ""
:global DNSFailCount; :set DNSFailCount 0
:global DNSRecoverCount; :set DNSRecoverCount 0
:global DNSFailoverAttempts; :set DNSFailoverAttempts 0
:global DNSMonitorRunning; :set DNSMonitorRunning false

The old DNSCurrentMode global from 0.1.0 is unused and can be unset.