healthcheck

Pre-Alpha. This page describes behavior that may change.

The Ze healthcheck monitors a service by running a shell command on a timer and controls BGP route announcement or withdrawal through a watchdog group. It is the successor to ExaBGP's healthcheck.py, integrated into the daemon instead of living alongside it.

The model is simple. You declare a probe. The probe runs a shell command every few seconds. After N consecutive successes, the probe is UP. After M consecutive failures, the probe is DOWN. Routes attached to the same watchdog group are either announced or withdrawn based on the state, or their MED is manipulated to shift traffic without withdrawing.

A minimal working probe

bgp {
    healthcheck {
        probe dns {
            command "dig @127.0.0.1 example.com +short";
            group hc-dns;
        }
    }
    peer upstream {
        connection { remote { ip 10.0.0.1; } }
        session    { asn { remote 65001; } }
        update {
            attribute { origin igp; next-hop self; }
            nlri      { ipv4/unicast add 10.0.0.1/32; }
            watchdog  { name hc-dns; withdraw true; }
        }
    }
}

The probe runs dig every 5 seconds (the default interval). After 3 consecutive successes, 10.0.0.1/32 is announced. After 3 consecutive failures, it is withdrawn. The bgp-healthcheck plugin depends on bgp-watchdog, and both load automatically when any healthcheck { } block is present.

Configuration reference

Every leaf goes under bgp { healthcheck { probe <name> { ... } } }.

Leaf	Type	Default	Description
command	string	required	Shell command. Exit 0 is success, non-zero is failure.
group	string	required	Watchdog group name. Must be unique across probes.
interval	uint32	5	Seconds between checks. 0 runs the check once and goes dormant.
fast-interval	uint32	1	Seconds between checks during RISING and FALLING.
timeout	uint32	5	Command timeout in seconds. Process group killed on timeout.
rise	uint32	3	Consecutive successes before UP.
fall	uint32	3	Consecutive failures before DOWN.
withdraw-on-down	bool	false	True: withdraw on DOWN/DISABLED. False: re-announce with down-metric / disabled-metric.
disable	bool	false	Admin disable. Probe enters DISABLED immediately.
debounce	bool	false	Only dispatch watchdog commands on state changes.
up-metric	uint32	100	MED value when UP.
down-metric	uint32	1000	MED value when DOWN (only if withdraw-on-down is false).
disabled-metric	uint32	500	MED value when DISABLED (only if withdraw-on-down is false).

Two modes

There are two ways to react when the probe goes DOWN, and picking between them is usually the first design decision.

Withdraw mode (withdraw-on-down true). Routes are announced when UP and withdrawn when DOWN or DISABLED. Traffic moves to whatever the upstream chooses as the next-best path. This is the right choice when you want traffic to actually go somewhere else.

MED mode (withdraw-on-down false, the default). Routes are always announced, but the MED changes. up-metric when UP, down-metric when DOWN, disabled-metric when DISABLED. Upstream routers prefer the lowest MED, so traffic shifts gracefully without a withdrawal. This is the right choice when you want soft failover rather than a hard withdrawal.

In withdraw mode, the route carries watchdog { name <group>; withdraw true; }. In MED mode, the route carries watchdog { name <group>; withdraw false; }. The withdraw toggle is evaluated per watchdog group, not per probe, so multiple probes feeding the same group compose cleanly.

Virtual IP management

In internal mode, a probe can add and remove virtual IPs on a local interface alongside the route changes.

probe dns {
    command "dig @127.0.0.1 example.com +short";
    group   hc-dns;
    ip-setup {
        interface lo;
        dynamic   false;
        ip        10.0.0.1/32;
        ip        10.0.0.2/32;
    }
}

With dynamic false, the IPs are added at probe startup and stay present across every state except EXIT. With dynamic true, the IPs are removed on DOWN and DISABLED and restored on UP. This feature is internal-mode only and is not available when the healthcheck runs as an external plugin.

Hooks

Probes can fire shell commands on state changes.

on-up       "curl -X POST http://notify/up";
on-down     "curl -X POST http://notify/down";
on-disabled "curl -X POST http://notify/disabled";
on-change   "logger -t healthcheck state=$STATE";

Each hook is a leaf-list, so multiple commands per event are allowed. Hooks run asynchronously with a 30-second timeout and a process-group kill on expiry, because runaway hook processes are a classic production hazard. $STATE is set to the current state name. State-specific hooks fire before on-change.

FSM states

State	Description
INIT	Initial state at startup.
RISING	Successes accumulating (count below rise).
UP	Service healthy. Routes announced with up-metric.
FALLING	Failures accumulating (count below fall).
DOWN	Unhealthy. Routes withdrawn, or re-announced with down-metric.
DISABLED	Admin disabled. Command not executed.
EXIT	Shutdown. Routes withdrawn, IPs removed.
END	Single-check complete (interval 0). Routes and IPs left in place.

CLI commands

Command	Description
healthcheck show	JSON summary of every probe.
healthcheck show <name>	Detailed status of a single probe.
healthcheck reset <name>	Withdraw the current route, reset the FSM to INIT, re-check immediately. Fails if the probe is DISABLED.

Migrating from ExaBGP

Ze's healthcheck replaces the ExaBGP healthcheck.py program. A few things change.

• Route attributes (communities, AS path, next-hop) are defined in the BGP config, not in the healthcheck config.
• MED override through a single watchdog group replaces per-state route definitions.
• Disabling a probe is ze config set ... disable true, not a file-poll mechanism.
• Hooks have a 30-second timeout. ExaBGP hooks could hang forever.
• Per-state community or AS-path variation is not supported. Use separate watchdog groups if you need it.

See also

• Operational reports for show warnings and show errors.
• Monitoring for the Prometheus endpoint that exposes probe state as a metric.
• BGP peers for the surrounding peer configuration.

Adapted from main/docs/guide/healthcheck.md.