Troubleshooting

Troubleshooting

Problem	Symptom	Fix
No trusted CA	DoH server connection error: SSL: no trusted CA certificate found (6)	import the CA, set trusted=yes
Wrong CA	error persists	Cloudflare = SSL.com TLS ECC Root CA 2022
Clock off	cert validation fails, monitor says "DoH down"	enable NTP (IP-based servers)
device-mode=home	not allowed by device-mode on fetch/scheduler	run the monitor by hand; monitor cannot probe there
DoH down	resolution stops, servers do not help	monitor switches to the fallback DoH
Sites do not open, DNS "fine"	CDNs time out, resolver healthy	stub pair — see below
Failover never triggers	log both endpoints unreachable	probe needs DNS: add bootstrap records

Fixed in 0.4.3 — the switch that never happened

Symptom	Cause	What changed
Primary endpoint is down, /ip dns still points at it, clients get nothing	In probeMode "resolve" the fallback was still probed over DNS-JSON; a self-hosted endpoint answers 400, so probeB never turned ok and the switch was skipped	In resolve mode the fallback is not probed — the monitor switches and verifies by :resolve afterwards, rolling back if the new endpoint answers badly
Substituted answers accepted on the fallback path	Only the primary probe looked at the payload	The fallback probe reports substituted as well

Reserve endpoint on a separate host — see "A reserve endpoint on a separate host" below.

A reserve endpoint on a separate host

A failover between two public resolvers is weak: both sit behind the same path and get substituted together. A reserve that helps is your own resolver on a host that is not on the broken path.

Shape: your existing web server (443) → location = /dns-query with allow/deny → an HTTP/1.1 DoH front end on loopback → unbound with forward-tls-upstream.

The pitfall. On Ubuntu 26.04 the packaged dnsdist is 2.x and its plain-HTTP DoH listener is HTTP/2-only: an HTTP/1.1 request gets an HTTP/2 frame back (BadStatusLine, 000), and nginx cannot proxy to it at all — nginx has no HTTP/2 upstream for proxy_pass. h2=false is rejected (Unknown key 'h2' given - ignored).

Options that work: dnsdist 1.7 (as on Debian 12), or a small gateway — http.server on loopback that turns a POST body into a UDP query and returns the answer. No dependencies, plain HTTP/1.1, easy to audit.

unbound cannot sit behind a plaintext proxy: http-endpoint needs TLS keys, and the packaged service usually cannot read a Let's Encrypt key without a copy and a deploy hook.

Checks, in order: gateway alone (200), through nginx from the host (400 on GET is correct — that is not a DoH request), from a client with the wire format (200 with honest addresses; 403 means the address is missing from the allow list).

Fixed in 0.4.2 — symptom, cause, what changed

Three field problems turned into code and documentation in this release. Each row names the symptom you would have seen, the cause, and where the fix lives.

Symptom	Cause	What changed
Clients open nothing for some names while the resolver reports healthy, Status: 0	The network substitutes the DoH answer itself: same URL, same certificate, stub address. The probe only looked at the status field	Probe checks the payload for stubPrefixes and reports substituted (see "The network substitutes DoH answers as well")
Own DoH endpoint shows as bad-status and the router jumps to a public fallback	The probe speaks DNS-JSON; a self-hosted resolver serves wire format and rejects that request	probeMode "resolve": liveness from live-answer quality, no external request
After pointing at an honest resolver, clients still get stubs for a while	The stubbed answer was already cached; it lives until TTL, so the monitor declared STUB on stale data	One cache flush and a re-check before declaring; STUB_CLEARED when the cache was the culprit
Importing the Zabbix modules fails with ZBX_USER is not set	The credential check ran at module level, so even import needed the environment	Check moved into main(); the console scripts read ZBX_USER/ZBX_PASSWORD when they run
Collector exits with a Python traceback	zabbix_sender was missing; subprocess raised FileNotFoundError	Clear message naming the binary and the package to install; path from ZBX_SENDER

How to verify each one on a live router: Verification → "0.4.2 checks".

Sites do not open while the resolver looks healthy

Symptom. cdn.tailwindcss.com, cdnjs.cloudflare.com, unpkg.com time out. The router is fine: DoH answers, Status:0, a valid A record.

What is returned. The pair 8.6.112.0 / 8.47.69.0. Measured 2026-10-26 from three vantage points:

Query	Answer
DoH, standard endpoint, Russian ISP	104.26.x at first, the pair ~30 minutes later
DoH, Family endpoint, same vantage	the pair
plain DNS to 1.1.1.1 and 8.8.8.8	the pair (Google answered 8.6.112.5)
the same questions from Germany	104.26.x, 172.66.x
curl --resolve example.com:443:8.6.112.0 from a gateway WAN	200, valid certificate
the same from another ISP's branch	timeout

The pair is served by Cloudflare and is a real answer — not a block, not a filter artefact. It is simply unreachable from some networks. Where the path is missing, every name resolving to it stops working, whatever endpoint you use.

Confirm from a client (the router cannot tell — /tool fetch cannot distinguish a dead path from an HTTP refusal):

curl -sS -o /dev/null --max-time 10 \
  --resolve cdn.tailwindcss.com:443:8.6.112.0 \
  -w '%{http_code} %{time_total}s\n' https://cdn.tailwindcss.com/
# 302 in ~0.2 s -> reachable here, the pair is harmless
# timeout      -> unreachable: this is your outage

What fixes it: route the pair through a working tunnel; or pin reachable addresses with /ip dns static for the names you need; or change the vantage point. Switching resolvers does not help, and /ip dns static type=FWD forward-to=1.1.1.1 does not either (the router forwards over plain DNS and gets the same pair).

The failover never triggers

The probe asks the endpoint by hostname. If the resolver is dead, that hostname cannot be resolved, the probe fails and the monitor concludes "both endpoints unreachable — keeping the current configuration". Pin bootstrap records for both endpoint hosts:

/ip dns static print where comment~"DoH bootstrap"

Not every endpoint speaks the JSON API

Endpoint	Probe URL
Cloudflare (all three)	https://cloudflare-dns.com/dns-query?name=example.com&type=A
Google	https://dns.google/resolve?name=example.com&type=A
Quad9	unsupported: DoH unable to decode BASE64-URL

Quad9 is fine as an upstream but cannot be probed this way — a Quad9 fallback would always look unreachable.

force-DNS does not cover clients that use the router

dstnat redirect catches clients that send DNS to other addresses. A client whose DNS points at the router is answered locally and never enters dstnat, so per-client resolver policies built on dst-nat do nothing for it. Check the client: ipconfig /all → DNS Servers.

Static DNS overrides everything

/ip dns static wins over the DoH answer — useful for split-horizon names and for pinning a reachable address; dangerous when a leftover entry keeps a dead address.

/ip dns static print
/ip dns cache print where name~"tailwindcss"

The network substitutes DoH answers as well (0.4.2)

A resolver can look healthy, answer Status: 0, and still hand out addresses that open nothing. The substitution is not always on plain DNS: measured on one network, the same DoH URL answered honestly when reached through an encrypted tunnel and returned the stub pair when reached from the router's own uplink. Same URL, same certificate, different answers.

It is also unstable: repeated queries alternate between honest and stubbed answers, so a single successful probe proves nothing.

Symptom	What it means
Status: 0 but the address is the stub pair	the path is intercepted; the answer is syntactically perfect
Alternating honest/stub answers for the same name	active substitution, not a cache problem
Public DoH fallback behaves exactly like the primary	both paths are intercepted; failover between operators cannot help
Filtering resolver ("family", 1.1.1.3) always returns the pair	it filters by design — never use it as a baseline

Since 0.4.2 the probe checks the payload against stubPrefixes and reports substituted instead of ok. STUB is declared only after a cache flush and a re-check (STUB_CLEARED when the stub came from the cache).

The reliable remedy is a path the network does not intercept: your own resolver on a host whose traffic is already inside a tunnel, published over DoH, used as dohUrlA with probeMode "resolve" (below). Read the stub prefixes from your own network — the default pair is one operator's example, not a universal value.

probeMode: own endpoint without a JSON API (0.4.2)

The probe speaks DNS-JSON (?name=…&type=A). A self-hosted endpoint serves wire format and answers that request with an error, so the monitor would call a healthy primary bad-status and move the router to a public fallback — the very path that gets substituted.

:local probeMode    "resolve"                        # judge by answer quality only
:local dohUrlA      "https://doh.example.net/dns-query"
:local probeUrlA    ""                               # unused in resolve mode

In resolve mode liveness is derived from the live answers: at least one sane answer and no stub prefixes means ok. Nothing external is fetched, so the probe also works when /tool fetch is restricted.

A 400/405 from the probe is not "DoH down"

POSTing to /dns-query without a valid body returns 400, GET returns 405. Both mean the TLS handshake and certificate validation succeeded — the endpoint is alive, only the request form was wrong.

:resolve details worth knowing

:put [:resolve example.com]                  # through the live resolver (DoH when set)
:put [:resolve example.com server=1.1.1.1]   # direct plain query, bypasses DoH

:resolve ... timeout= is not supported (bad parameter timeout).