Remediation, State, and Exit Codes

Watchdog separates detection from remediation. A failed health check does not automatically mean a restart is executed: Watchdog first evaluates the state transition, then considers whether remediation commands exist and whether the service-specific cooldown permits an action. This separation reduces duplicate alerts and prevents a short scheduler interval from continually restarting the same service. 1

Execution flow

The following diagram describes the normal path for one enabled service. A successful check ends the service run unless it represents recovery from a previously stored unavailable state. 1

flowchart TD
    A[Run configured check] --> B{Check succeeds?}
    B -->|Yes| C[Set action status: not-required]
    C --> D{Previous state unavailable?}
    D -->|Yes| E[Send recovery notification and hooks]
    D -->|No| F[Record healthy state]
    E --> F
    B -->|No, attempts remain| A
    B -->|No, all attempts failed| G[Handle failure transition: notify, run hooks, and record unavailable state]
    G --> H{Remediation commands and cooldown allow?}
    H -->|No| I[Log unavailable]
    H -->|Yes| J[Record action attempt and run commands sequentially]
    J --> K[Wait verify_after, then re-run full check]
    K -->|Recovered| L[Record healthy state and recovery transition]
    K -->|Still failing| I

Loading

Failure notifications occur before remediation. This records the incident even when the corrective action succeeds quickly, while state tracking prevents duplicate failure notifications during the same continuing outage. 1

Retry and remediation controls

Control	Where it is set	Meaning
check.attempts	Service check map	Number of times to attempt the entire health check before treating it as unavailable.
check.retry_delay	Service check map	Seconds to wait between failed attempts; 0 disables the wait.
actions.commands	Service actions map	Ordered direct command arrays. The sequence stops at the first failure.
actions.cooldown	Service actions map	Minimum seconds between remediation attempts for that service. 0 allows an action on every run.
actions.verify_after	Service actions map	Seconds to wait after remediation commands before the full verification check. 1

The cooldown is tracked per service in the state directory. When a remediation action is due, Watchdog records the timestamp before executing the commands, so an interrupted or failed action is still subject to the configured cooldown on the next run. 1

Persistent state and transition behavior

For each service, Watchdog reads <state_directory>/<service>.state. Valid stored values are healthy and unavailable; an absent or invalid file is interpreted as unknown. It writes state atomically through a temporary file and records the last remediation timestamp in <state_directory>/<service>.last-action. 1

Previous state	New result	State update	Email and hook behavior
unknown	Healthy	Writes healthy	No recovery notification or recovery hook.
unknown	Unavailable	Writes unavailable	Sends failure email and runs failure hooks, if configured.
healthy	Healthy	No transition event	No repeated notifications or hooks.
healthy	Unavailable	Writes unavailable	Sends one failure email and runs failure hooks.
unavailable	Unavailable	No transition event	Does not resend a failure email or rerun failure hooks. Remediation may still recur after cooldown.
unavailable	Healthy	Writes healthy	Sends one recovery email and runs recovery hooks. 1

If a state reset is genuinely necessary—for example, while testing a new environment—stop the scheduler and remove only the relevant <service>.state and, if required, <service>.last-action files. Understand that the next observed failure will then be treated as a new unknown → unavailable transition.

Action status in notifications

The {{action_status}} mail-template variable captures Watchdog's remediation context. Common values include those below. The failure notification is rendered before commands run, so an action that is due is initially reported as pending. 1

Value	Meaning
not-required	The check succeeded; no remediation was needed.
not-configured	The check failed and the service has no remediation commands.
skipped-dry-run	A failure occurred during dry-run mode.
pending	Remediation commands are configured and allowed by the cooldown.
cooldown	Commands are configured but a previous action is still within the cooldown window.
commands-succeeded	Commands returned zero; a verification result is still pending.
command-failed	At least one remediation command failed.
successful	Post-remediation verification succeeded.
verification-failed	Commands succeeded but the verification check still failed. 1

Locking, dry runs, and filtering

Watchdog holds a global non-blocking flock for the duration of a run. If another invocation already has the lock, the new run logs action=lock result=already-running and exits with code 0. This makes an overly frequent scheduler safer, but a skipped run is a signal to review command timeouts and schedule cadence. 1

Use -n for a dry run. Dry run still validates configuration, acquires the lock, runs health checks, and writes operational logs, but it skips remediation, hooks, state reads/writes, and notification transitions. Use -s SERVICE to process exactly one configured service; a non-existent name is a configuration error. 1

# Validate and check one service without changing state or remediating it.
sudo /opt/service-watchdog/service-watchdog.sh \
  -c /etc/service-watchdog/config.yaml \
  -s public-api \
  -n

Exit codes

Exit code	Meaning	Scheduler interpretation
0	All selected services were healthy, no remediation was attempted, or the run was skipped because the global lock was held.	Normal successful invocation.
1	At least one selected service remains unavailable or remediation was attempted, even when verification subsequently confirms recovery.	Expected monitoring outcome; the supplied systemd unit treats it as successful.
2	Configuration, dependency, environment, or option error.	Investigate immediately; the supplied systemd unit marks this as a failure. 1 2

Next, configure Notifications and Hooks for transition-aware alerting and external integrations.

References