Notifications and Hooks

Watchdog can send built-in SMTP email and run local hook commands. Both mechanisms are transition-based: a service that stays unavailable does not produce the same failure notification on every scheduler run. Failure handling occurs when a service changes to unavailable; recovery handling occurs only when a previously unavailable service becomes healthy. 1

Event	When it occurs	Built-in email	Hook sequence
Failure	unknown or healthy changes to unavailable after all attempts fail.	Uses notifications.email.failure.	Uses hooks.on_failure.
Repeated outage	unavailable remains unavailable.	Not resent.	Not rerun.
Recovery	unavailable changes to healthy.	Uses notifications.email.recovery.	Uses hooks.on_recovery. 1

Alert transitions describe availability, not whether remediation ultimately succeeds. Watchdog records and notifies an outage before remediation; if the post-remediation verification is successful, it then sends one recovery notification. 1

Built-in SMTP email

Set notifications.email.enabled: true and provide the complete SMTP configuration. smtps:// is appropriate for implicit TLS on port 465; smtp:// can be used for SMTP upgraded with STARTTLS, for example on port 587. With the default tls_required: true, Watchdog requires a secure SMTP connection. 1 2

notifications:
  email:
    enabled: true
    smtp:
      url: smtps://smtp.example.com:465
      from: watchdog@example.com
      username: watchdog@example.com
      password_env: WATCHDOG_SMTP_PASSWORD
      tls_required: true
      insecure_skip_verify: false
      timeout: 30
    recipients:
      - administrator@example.com
      - on-call@example.com
    failure:
      subject: "[watchdog] {{service}} is unavailable"
      body: |-
        Watchdog detected a service availability problem.

        Service: {{service}}
        Time: {{timestamp}}
        Check type: {{check_type}}
        Detail: {{detail}}
        HTTP status: {{http_status}}
        Check exit code: {{check_exit}}
        Remediation: {{action_status}}
    recovery:
      subject: "[watchdog] {{service}} recovered"
      body: |-
        Watchdog confirmed that the service is available again.

        Service: {{service}}
        Time: {{timestamp}}
        Check type: {{check_type}}
        Detail: {{detail}}
        Remediation: {{action_status}}

SMTP field reference

Field	Required when enabled	Default	Rules and recommendation
smtp.url	Yes	None	smtp:// or smtps:// URL with no spaces.
smtp.from	Yes	None	One sender email address; used for the envelope and From header.
smtp.username	No	Empty	Optional, but it must be present when a password is provided.
smtp.password_env	No	Empty	Name of the environment variable containing the password. Preferred over inline secrets.
smtp.password	No	Empty	Inline password fallback. Do not set it together with password_env.
smtp.tls_required	No	true	Requires an encrypted SMTP connection. Keep enabled for Internet-facing servers.
smtp.insecure_skip_verify	No	false	Disables certificate verification. Use only with a deliberately trusted self-signed server.
smtp.timeout	No	30	Integer from 1 through 60 seconds; limits both SMTP connection and whole request.
recipients	Yes	None	Non-empty YAML array of email addresses. Each email is delivered to every recipient.
failure.subject, recovery.subject	Yes	None	Single-line strings.
failure.body, recovery.body	Yes	None	String or YAML multiline block. Messages are UTF-8. 1 2

Watchdog validates the password_env variable-name format and refuses a configuration that contains both password fields. It also refuses a username without a password and a password without a username. 1

Store credentials outside YAML

For the supplied systemd service, create a root-owned environment file. The unit reads /etc/service-watchdog/environment on every one-shot service start, so a timer restart is not required after changing the secret. 2 3

sudo install -m 0600 /dev/null /etc/service-watchdog/environment
sudoedit /etc/service-watchdog/environment
sudo chmod 0600 /etc/service-watchdog/environment

Add the variable named by password_env:

WATCHDOG_SMTP_PASSWORD=replace-with-the-real-password

When using cron instead, place the variable in the protected execution environment or use a dedicated secret-management mechanism. Do not commit or make world-readable a configuration that contains passwords, tokens, or notification endpoints. 2

Template variables

Watchdog substitutes the following placeholders in the subject and body before sending a notification. {{detail}} is sanitized and bounded before insertion. 1

Variable	Value
{{service}}	Service name from services[].name.
{{event}}	failure or recovery.
{{timestamp}}	Local date, time, and UTC offset at mail creation.
{{check_type}}	http, tcp, or command.
{{detail}}	Diagnostic detail from the latest check.
{{http_status}}	HTTP response code, or n/a for non-HTTP checks.
{{check_exit}}	Latest check exit code, or n/a when unavailable.
{{action_status}}	Remediation state such as pending, cooldown, or not-configured. 1

If SMTP delivery fails, Watchdog logs result=email-failed and still records the state transition. This avoids retrying the same failure email on every run; monitor the operational log so a mail-delivery failure is not missed. 1 2

Hooks and local integrations

Use hooks for integrations that are better handled by a local executable, such as an incident-management bridge, a chat webhook wrapper, or a custom mailer. Hooks use the same direct command-array structure as remediation commands and run only on transitions. 1

hooks:
  on_failure:
    - command: [/usr/local/bin/notify-watchdog]
      timeout: 30
  on_recovery:
    - command: [/usr/local/bin/notify-watchdog]
      timeout: 30

Each hook command inherits these environment variables. Keep sensitive integration values outside YAML so the hook script can obtain them from a controlled environment file or secret manager. 1 2

Variable	Failure value	Recovery value
WATCHDOG_SERVICE	Service name	Service name
WATCHDOG_EVENT	unavailable	healthy
WATCHDOG_DETAIL	Latest check diagnostic	Latest check diagnostic
WATCHDOG_CHECK_TYPE	Check type	Check type
WATCHDOG_HTTP_STATUS	HTTP status or empty	HTTP status or empty
WATCHDOG_CHECK_EXIT	Check command exit code or empty	Check command exit code or empty
WATCHDOG_TIMESTAMP	Event command timestamp	Event command timestamp 1

A hook failure is logged but does not stop the remaining Watchdog state update. Design hook scripts to be idempotent and fast, set explicit timeouts, and return meaningful non-zero codes on failure.

Test notifications safely

Validate first with dry run, then use a controlled test service to create a real state transition. Starting Watchdog while every service remains healthy validates configuration but does not send a test email, because no failure or recovery transition occurred. 2

sudo /opt/service-watchdog/service-watchdog.sh \
  -c /etc/service-watchdog/config.yaml \
  -n

sudo systemctl start service-watchdog.service
sudo journalctl -u service-watchdog.service -n 50 --no-pager
sudo tail -n 50 /var/log/service-watchdog/service-watchdog.log

Continue with Scheduling with systemd and cron to make these transition checks run automatically.

References