Service checks

Service checks

A service check tells you which level of a service is down. Add one [[services.checks]] block to config.toml for each service:

[[services.checks]]
name = "Jellyfin"
container = "jellyfin"
url = "http://192.0.2.10:8096/health"
expect = "Healthy"
vpn_url = "http://server.tailnet.example:8096/health"
public_url = "https://jellyfin.example.com"

[[services.checks]]
name = "SSH"
tcp = "192.0.2.10:22"
public_tcp = "home.example.com:22"
every = 300

Field	Description
name	Required. The name in the Services table.
container	The Docker container name. The check needs [host] docker_proxy.
url or tcp	The Local stage: a local http:// or https:// URL, or a TCP target.
vpn_url or vpn_tcp	The VPN stage, for an address on your VPN, for example Tailscale or WireGuard.
public_url or public_tcp	The Public stage: the public URL through your reverse proxy, or a public TCP target.
expect	Text that the reply must contain.
every	Seconds between checks of this service. The default is 60. The minimum is 1 for a service with a container, and 60 for a service with no container.

Each block needs a name and at least one container, URL, or TCP target. Each stage takes either a URL or a TCP target. A TCP target is host:port, or [address]:port for an IPv6 address.

The bot sends the requests from inside its own container, so localhost is the bot container. Use the IP address of the server, or a host name that resolves inside the container.

To skip a service and keep its block, add its name to [services] ignore (see Configuration).

Stages

The bot runs the checks every 60 seconds, or at the every time of each service. It does the stages of a service in this order:

1. Container: the container must exist, run, and not report unhealthy.
2. Local: the bot sends a GET request to url, or opens a connection to tcp.
3. VPN: the same, for vpn_url or vpn_tcp.
4. Public: the same, for public_url or public_tcp.

The bot skips a stage that has no target. After a stage fails, the bot does not run the stages after it. As a result, the first failure names the level that is down. expect applies to the first stage that has a URL.

A URL stage passes on a 1xx, 2xx, or 3xx status, and on 401 or 403. The bot does not follow redirects, so a 3xx shows that the server answered. A 401 or 403 shows that the proxy and its login page are up. The timeout is 8 seconds, and 4 seconds for the connection. A TCP stage passes when the connection opens, and its timeout is 4 seconds.

The table

The table has these columns: Service, Status, then one column for each stage. [services] columns sets the labels of the stage columns. The default labels are Local, VPN, and Public.

Services with a container appear in the Services table. Services with no container appear in a second table, under the heading External Checks. The text mode leaves out a stage column when no service uses that stage. It also leaves out columns that do not fit a phone screen.

The Status column shows the container stage:

Value	Meaning
ok	The container runs. For a service with no container, all stages passed or only had warnings.
starting	The container health check did not finish yet. Warning.
unhealthy	The container health check failed. Critical.
missing	No container has that name. Critical.
a state, for example exited (1)	The container does not run. The number is the exit code. Critical.
error	A service with no container failed a critical stage.

The stage columns show these values:

Value	Meaning
a time, for example 42ms	The stage passed in that time.
CF wall	Cloudflare answered with a bot challenge. The stage passes, but the bot cannot see your server behind Cloudflare.
-	An earlier stage failed, so this stage did not run.
N/A	No target is set for this stage.
no relay	Public column only. No relay answered, so the bot could not do the check. Warning.
timeout	No reply in time.
DNS	The host name did not resolve.
TLS	The HTTPS connection failed, for example because of a bad certificate.
refused	The server refused the connection. Nothing listens on that port.
no route	The bot could not connect to the host.
bad reply	The reply was not valid HTTP.
HTTP 502	The server answered with that error status.
bad body	The reply did not contain the expect text. The bot reads the first 64 KiB of the reply only.
not allowed	Relay only. The host is not on RELAY_ALLOW.
not public	Relay only. The host resolves to an address that is not a public internet address.
closed	Relay only. The TCP connection failed.

A failed container stage is critical. A failed URL or TCP stage is critical when no URL or TCP stage before it passed, and a warning when one did. A healthy container does not count as a passed stage. A stage that takes more than 1500 ms gives a slow warning. Each problem also appears in the alert list, for example Jellyfin local refused.

Relay

A relay is an optional copy of Shazzzbot that runs on a machine outside the network of the server. With a relay, the bot sends each Public stage to the relay. The check then uses the same path as a visitor from the internet.

1. On the other machine, run bash setup.sh --role relay (see Installation). The first start prints RELAY_URL, RELAY_KEY, and RELAY_FINGERPRINT.
2. Put those three lines in the .env file of the bot. In RELAY_URL, replace <this machine's address> with an address of the relay machine that the bot can reach.
3. Recreate the bot container.

The relay listens on port 8443 on all addresses of its machine. To use one address only, set RELAY_BIND to that address. The relay keeps its key and certificate in the data directory of its install. Later starts print only RELAY_FINGERPRINT.

These variables in the .env file of the relay control it:

Variable	Description
RELAY_ALLOW	Required. The host names or IP addresses that the relay may check, separated by spaces or commas.
RELAY_BIND	The host address for port 8443. The default is 0.0.0.0, that is all addresses.
RELAY_KEYS	More API keys that the relay accepts, separated by spaces. Make each one with openssl rand -hex 16.

The relay checks only names on RELAY_ALLOW and their subdomains, and only when each address that a name resolves to is a public internet address. The relay accepts only TLS 1.3. The bot checks the relay certificate against RELAY_FINGERPRINT before it sends the key, and refuses a relay that does not match.

After 3 wrong keys in 5 minutes from one address, the relay refuses that address for 24 hours. For an address that used the right key before, the limit is 20 wrong keys. After 30 refusals in 5 minutes, the relay serves only addresses that used the right key before, until 1 hour passes with no such flood. A restart of the relay removes these blocks.

For more relays, add the same three names with _2, _3, and so on at the end. A Public stage passes when one relay or more can reach the target.

The System section shows the relays in its node status. With a relay, the [services] public_host setting also lets the bot list the published ports that the internet can open, under Open to the internet (see Configuration). The name in [services] public_host must be on RELAY_ALLOW.