Examples and Recipes

The repository contains ready-to-adapt configuration files for common deployments. Copy the closest example to a root-owned location, replace placeholder domains, paths, ports, service names, and commands, then run it in dry-run mode. The example paths are intentionally illustrative: configuration validation fails until every referenced working_directory exists on the target host. 1

Repository example	Use case	Start here
http-docker-compose.yaml	HTTP health endpoint with Docker Compose restart.	HTTP and HTTPS checks
tcp-systemd.yaml	TCP port check with systemd restart.	TCP checks
command-check.yaml	Local command health check.	Command checks
multiple-services.yaml	HTTP, TCP, and command checks in one run.	Services
hooks.yaml	Failure and recovery integration hooks.	Hooks and local integrations
smtp-email.yaml	Built-in transition-based SMTP email.	Built-in SMTP email

1. HTTP endpoint with Docker Compose remediation

Use this pattern for an API that exposes a health endpoint and is managed by Docker Compose. The health signal is the HTTP response, while remediation restarts the relevant Compose services from the directory containing the Compose file. This separates a client-visible readiness test from the restart action. 2

services:
  - name: api
    check:
      type: http
      url: https://api.example.com/health
      method: GET
      follow_redirects: true
      success_status: [200, 204]
      timeout: 10
      attempts: 2
      retry_delay: 2
    actions:
      cooldown: 300
      verify_after: 5
      commands:
        - command: [docker, compose, restart, api]
          working_directory: /srv/example-api
          timeout: 120
        - command: [docker, compose, restart, nginx]
          working_directory: /srv/example-api
          timeout: 120

Test it without changing state or restarting containers:

sudo /opt/service-watchdog/service-watchdog.sh \
  -c /etc/service-watchdog/config.yaml \
  -s api \
  -n

Ensure that the scheduler account can access Docker and that /srv/example-api exists. On the packaged unit, Watchdog runs as root, but a custom service account may need suitable socket or group access. 2 3

2. TCP database listener with systemd restart

Use a TCP check when service availability means that a known host and port must accept connections. The recipe below monitors a local PostgreSQL listener and gives the service three attempts before starting remediation. It confirms recovery after a three-second delay. 4

services:
  - name: local-postgresql
    check:
      type: tcp
      host: 127.0.0.1
      port: 5432
      timeout: 10
      attempts: 3
      retry_delay: 2
    actions:
      cooldown: 300
      verify_after: 3
      commands:
        - command: [systemctl, restart, postgresql]
          timeout: 60

A TCP success means only that a connection opened. If the database can accept TCP connections while still failing application queries, use a command check that runs an appropriate local probe instead. 3

3. Command health check for a systemd worker

Use a command check when the local host already has a reliable service-manager signal. The command array below avoids shell parsing and checks the worker's systemd state. 5

services:
  - name: background-worker
    check:
      type: command
      attempts: 1
      commands:
        - command: [systemctl, is-active, --quiet, example-worker]
          timeout: 30
    actions:
      cooldown: 300
      verify_after: 2
      commands:
        - command: [systemctl, restart, example-worker]
          timeout: 60

For an application-specific command, preserve argument boundaries. For example, write command: [/usr/local/bin/check-queue, --minimum-workers, "2"], not a single shell string containing spaces or redirection syntax. 3

4. Monitor several services in one configuration

A single run may process HTTP, TCP, and command services in the order they appear in services. Each service retains its own state and cooldown files, so a continuing outage in one service does not suppress checking or remediation of the others. 3 6

services:
  - name: public-api
    check:
      type: http
      url: https://api.example.com/health
      success_status: [200, 204]
    actions:
      cooldown: 300
      verify_after: 5
      commands:
        - command: [systemctl, restart, example-api]

  - name: redis
    check:
      type: tcp
      host: 127.0.0.1
      port: 6379
    actions:
      cooldown: 180
      verify_after: 2
      commands:
        - command: [systemctl, restart, redis-server]

  - name: queue-worker
    check:
      type: command
      commands:
        - command: [systemctl, is-active, --quiet, queue-worker]
    actions:
      cooldown: 300
      verify_after: 2
      commands:
        - command: [systemctl, restart, queue-worker]

During configuration work, target one service with -s to keep testing narrow. When you are ready to validate the combined configuration, omit -s but retain -n for the first complete run. 1 3

5. Add email and hooks deliberately

Email and hooks should be added after the base check is known to work. The next example enables SMTP with an environment-supplied password and introduces a local notification script. Both mechanisms execute on state transitions only. 7 8

notifications:
  email:
    enabled: true
    smtp:
      url: smtps://smtp.example.com:465
      from: watchdog@example.com
      username: watchdog@example.com
      password_env: WATCHDOG_SMTP_PASSWORD
      tls_required: true
      insecure_skip_verify: false
      timeout: 30
    recipients: [ops@example.com]
    failure:
      subject: "[watchdog] {{service}} is unavailable"
      body: "Check failed: {{detail}}"
    recovery:
      subject: "[watchdog] {{service}} is healthy"
      body: "Recovered at {{timestamp}}."

hooks:
  on_failure:
    - command: [/usr/local/bin/notify-watchdog]
      timeout: 30
  on_recovery:
    - command: [/usr/local/bin/notify-watchdog]
      timeout: 30

Create the secret in /etc/service-watchdog/environment with mode 0600 when using the packaged systemd unit. Then conduct a controlled outage-and-recovery test to verify the actual message delivery and integration behavior. See Notifications and Hooks for the full template-variable list and secret-handling guidance. 7 9

Recipe validation sequence

Stage	Command or action	Expected result
Schema validation	service-watchdog.sh -c CONFIG -n	YAML, paths, command arrays, and dependency checks pass.
Isolated service test	Add -s SERVICE -n	Only one selected service is checked; no state or remediation changes occur.
Full dry run	Remove -s, keep -n	All services are checked in configuration order.
Controlled incident	Temporarily make a non-production target fail.	Confirms state change, alerting, remediation, and verification.
Scheduled validation	Enable timer or cron entry.	Logs show periodic invocations with expected exit semantics. 1 3

For operational diagnosis after deployment, continue with Operations and Troubleshooting.

References