Home

Watchdog

Watchdog is a small, dependency-light Bash program for checking websites and services and running an explicit remediation sequence when a target remains unavailable after its configured retries. It is deliberately a one-shot process: pair it with a systemd timer, cron, or another scheduler rather than running it as a permanent daemon. The project supports HTTP/HTTPS endpoint checks, TCP connection checks, and command-based checks, with persistent state to make alerting transition-aware. 1

Design principle: configured commands are YAML argument arrays, not shell strings. Watchdog invokes them directly and does not use eval or bash -c for configured commands. 2

Start here

A typical production deployment follows the sequence below. Begin with a dry run, confirm that every configured path and command exists on the host, then enable a scheduler only after the checks behave as expected.

Step	Outcome	Wiki page
1	Install Bash, curl, GNU timeout, flock, and Mike Farah yq v4.	Installation and First Run
2	Create a YAML configuration with one or more services.	Configuration Reference
3	Choose the correct HTTP, TCP, or command health check.	Check Types
4	Define safe remediation, retries, verification, and cooldowns.	Remediation, State, and Exit Codes
5	Add transition-only email notifications or local hooks.	Notifications and Hooks
6	Schedule the one-shot check with the packaged timer or cron.	Scheduling with systemd and cron
7	Validate logs, state, permissions, and recovery behavior.	Operations and Troubleshooting

What Watchdog does

Watchdog evaluates every enabled service in the YAML services array. A check may succeed immediately or retry according to its attempts and retry_delay settings. If all attempts fail, Watchdog records an unavailable transition, sends any configured failure notification once, and—subject to a per-service cooldown—runs the ordered remediation commands. It can wait for verify_after seconds and then run the full check again to confirm recovery. 2

Capability	Practical value
HTTP/HTTPS monitoring	Verify a health endpoint with GET or HEAD, optional redirect following, a timeout, and either default 2xx success or an explicit status allow-list.
TCP monitoring	Confirm that a host and port accept a TCP connection within the configured timeout.
Command monitoring	Use local commands, such as systemctl is-active --quiet, as a health signal.
Ordered remediation	Run direct command arrays sequentially and stop the sequence at the first non-zero exit status.
State-aware notifications	Avoid alert floods: failure and recovery notifications occur on relevant state transitions rather than every scheduled run.
Safe scheduled operation	Use a non-blocking global lock to avoid overlapping runs when a previous invocation is still active. 2

Concepts and terminology

Term	Meaning
Check	The configured test that determines whether a service is healthy.
Attempt	One execution of the check. A service can have multiple attempts before being declared unavailable.
Remediation	An optional, ordered command sequence intended to correct an unavailable service.
Verification	A new full health check performed after remediation, optionally delayed by verify_after.
State	The persisted status healthy or unavailable; an absent or invalid state file is treated as unknown. 2
Cooldown	The minimum number of seconds between remediation attempts for the same service.
Hook	A local command sequence triggered only when the service changes state.

Version scope and source of truth

This Wiki is written for Watchdog v1.0.6. The executable, its example configuration, installer, systemd units, and tests remain the authoritative description of behavior. If a later release changes the schema or runtime behavior, update the affected Wiki page together with the release notes. 1 2

For complete, ready-to-adapt YAML files, see Examples and Recipes. For day-two operation, alert diagnosis, and security hardening, continue with Operations and Troubleshooting.

References