Monitoring

Monitoring the fleet in Zabbix

The scripts in routeros/ keep each router healthy on its own. This page covers the other half: seeing the state of the whole fleet — DoH and VPN — in one place, with triggers that fire before users call.

Everything below is shipped in the repository under zabbix/ — note that zabbix/ was committed after the v0.2.0 tag, so it is on main and in no release yet. Measured on Zabbix 8 against six MikroTik devices (RouterOS 7.23.1 and 7.24.5 — the two versions this repository records).

---

Why a collector instead of Zabbix agents

A Zabbix agent on the router is not an option: the fleet includes devices where device-mode is home, so fetch and scheduler are closed. Even on advanced devices, adding an agent means another item of software to keep alive.

So the direction is reversed. A single collector on the management host (192.168.31.31 in our deployment) opens one SSH session per router every two minutes, asks a few questions with RouterOS scripting, and pushes the answers to Zabbix as trapper items.

management host
  doh-zabbix-collector.timer  ──▶  doh-zabbix-collector.service
        │  ssh chateau-md@<router>   (key /root/.ssh/id_mikrotik_agent)
        ▼
  RouterOS:  :resolve three canaries
             /ip dns get use-doh-server
             /interface wireguard peers  (count, last-handshake)
        │
        ▼  zabbix_sender 127.0.0.1:10051
  Zabbix trapper items ──▶ triggers ──▶ service tree

One command per router, sent as a single SSH argument (no intermediate shell), so there is no quoting to get wrong:

:local r "";
:foreach d in={"example.com";"cdnjs.cloudflare.com";"cdn.tailwindcss.com"} do={
  :local ip ""; :do {:set ip [:tostr [:resolve $d]]} on-error={:set ip ""};
  :set r ($r . $d . "=" . $ip . ";")};
:put ("EP=" . [/ip dns get use-doh-server]);
:put ("MON=" . [:len [/system scheduler find name="Scheduler_DNS_Health_Check"]]);
:put ("CAN=" . $r);
:put ("CLK=" . [/system clock get date]);
:local n 0; :local m 999999;
:foreach p in=[/interface wireguard peers find] do={
  :set n ($n + 1); :local hs [/interface wireguard peers get $p last-handshake];
  :if (($hs != "") and ($hs > 0) and ($hs < $m)) do={:set m $hs}};
:put ("PEERS=" . $n); :put ("HS=" . $m);
:put ("GW=" . [:len [/ip route find where routing-table="gw535" and active=yes and dst-address="0.0.0.0/0"]])

All six :put lines matter: the collector parses EP, MON, CAN, CLK, PEERS, HS and GW, and a line missing from the snippet leaves its item empty (doh.monitor never builds without MON, for example).

---

Items

Key	Type	Meaning
doh.state	numeric	0 healthy, 1 stub pair, 2 not answering, 3 no data
doh.endpoint	text	current /ip dns use-doh-server
doh.canary	text	answers for the three canaries
doh.monitor	numeric	does the Scheduler_DNS_Health_Check entry exist on the device (1/0)
vpn.state	numeric	0 ok, 1 degraded, 2 down, 3 no data
vpn.peers	numeric	number of WireGuard peers
vpn.handshake	numeric (s)	age of the freshest handshake
vpn.detail	text	the same in words

doh.state is decided by content, exactly like the on-device monitor: any canary resolving to a real address means healthy; only stub answers mean stub; no answer at all means dead.

vpn.state thresholds: no peers → down; freshest handshake older than 15 minutes → down; older than 5 minutes → degraded. Devices that carry no tunnels are pinned to ok — they are not supposed to have handshakes.

---

Triggers

Name	Expression	Severity
DoH is down	last(/<host>/doh.state)=2	Disaster
No DoH data	nodata(/<host>/doh.state,15m)=1	Warning
VPN tunnel is down	last(/<host>/vpn.state)=2	Disaster
VPN tunnel is unstable	last(/<host>/vpn.state)=1	Warning
No VPN data	nodata(/<host>/vpn.state,15m)=1	Warning

There is deliberately no trigger for the stub pair. For the networks this fleet sits in, 8.6.112.x / 8.47.69.x is the normal Cloudflare answer for legitimate names — see Troubleshooting §6.10. A trigger on it would be open forever and would train everyone to ignore the board. The condition stays visible in doh.state (=1) and in doh.canary, where it can be read when a site misbehaves.

---

Service tree

DoH на микротиках                 («DoH on the MikroTiks»)
  └── DoH на микротиках / mt-31-1     (problem tags: service=doh, device=mt-31-1)
  └── … one per device

VPN
  ├── Transport, Tunnel, Routing, …   (pre-existing children of the VPN root)
  └── Микротики                    («MikroTiks»)
        └── Микротики / mt-31-1      (problem tags: service=vpn, device=mt-31-1)
        └── … one per device

The service names are Cyrillic on purpose — they are the strings the script creates, and the Zabbix UI shows them exactly as written. Look for them by name, not by a translated label.

Open it at https://<zabbix>/zabbix.php?action=service.list.

Four Zabbix 8 behaviours worth knowing before building the same thing:

• a service cannot have problem_tags and children at the same time — parents aggregate children, leaves take their status from tags;
• the script creates every service with algorithm=1, which means "problem if at least one child has a problem" — the behaviour you want, and the one measured on 2026-10-26, where a single device's problem reached «Микротики» and then «VPN». algorithm=2 means "problem only if all children have problems", i.e. one dead site would stay invisible;
• status = -1 in the API means OK, not "unknown". Test for status >= 1;
• parentid and the auth parameter are gone: the token travels in the Authorization header, and hierarchy is an array parents. Value maps belong to a host, not to the system.

---

Install

# 1. objects in Zabbix (idempotent - safe to re-run after edits)
export ZBX_USER=... ZBX_PASSWORD=...    # API login; it is not stored in the file
python3 doh-zabbix-setup.py

# 2. collector and timer on the management host
install -m 755 doh-zabbix-collector.py /usr/local/bin/
cp systemd/doh-zabbix-collector.{service,timer} /etc/systemd/system/
systemctl daemon-reload && systemctl enable --now doh-zabbix-collector.timer

# 3. verify
systemctl list-timers doh-zabbix-collector.timer
journalctl -u doh-zabbix-collector.service -n 10

The setup script finds the routers by the addresses of existing hosts, so it reuses mt-31-1, mt-1-1 and friends instead of creating duplicates. Hosts are matched on the value in the item's interface, which means renaming a host in Zabbix does not break the mapping.

Prerequisites: zabbix_sender, an SSH key with chateau-md@ access on every router, and current host keys in known_hosts. A stale RSA entry in known_hosts is the most common reason the collector reports unknown for one device.

---

Proving it works

Sending a value is not the same as a working trigger chain. To check the chain, stop the timer first — otherwise it overwrites the injected value two minutes later and the test proves nothing:

systemctl stop doh-zabbix-collector.timer
printf '"mt-31-1" vpn.state "2"\n' | zabbix_sender -z 127.0.0.1 -i -

Expected: the problem "VPN tunnel is down" opens, MikroTiks / mt-31-1 goes to Disaster, and within a minute MikroTiks and VPN follow. Then restore:

printf '"mt-31-1" vpn.state "0"\n' | zabbix_sender -z 127.0.0.1 -i -
systemctl start doh-zabbix-collector.timer

Measured on 2026-10-26: problem opened at 14:25:23 and closed at 14:25:27; aggregation child → MikroTiks → VPN confirmed with the timer stopped.

---

Limits

• Resolution is two minutes. A tunnel that drops and recovers inside one interval is invisible here — that is what the on-device monitor and its log are for.
• One SSH hop per router. If the management host cannot reach a router, that device reports no data and the nodata() trigger fires; it does not mean the router is broken.
• last-handshake is a duration, not a number. RouterOS returns 00:01:00; the collector parses w/d/h:m:s and plain seconds. Sending the raw string leaves the item at zero.
• The collector never changes anything on a router. It only reads and resolves.

Install the tools as a package

The RouterOS side stays copy-by-hand: a router configuration is reviewed, not installed. The Zabbix side is ordinary Python, so it ships as a package:

pip install mikrotik-doh-failover
doh-zabbix-setup       # create host, items and triggers
doh-zabbix-collector   # send one round of values

Credentials are read at run time from ZBX_USER / ZBX_PASSWORD; the sender path from ZBX_SENDER. A missing sender now says what to install instead of raising a traceback. Installing is optional — the same files run from a checkout.