Scheduling with systemd and cron

Watchdog is a one-shot program. It performs the configured checks, optional remediation, and state handling once, then exits. A scheduler is therefore required for continuous monitoring. The repository ships a hardened systemd oneshot service and a persistent timer; cron is a supported alternative when it better fits the host environment. 1 2

Scheduler	Prefer it when	Main advantage
Packaged systemd timer	The host uses systemd and remediation may need root privileges.	Integrates with the journal, a protected environment file, persistence, and the supplied unit hardening.
Root crontab	The deployment standard is cron or systemd is not available.	Simple and portable periodic invocation.
Another scheduler	A platform scheduler already owns job execution.	Use the same command line and preserve the required permissions and environment.

Use the packaged systemd timer

After installation and a successful dry run, enable the timer. The packaged timer begins one minute after boot, then runs the service every minute with a ten-second accuracy window; Persistent=true allows systemd to account for missed intervals across downtime. 1

sudo systemctl enable --now service-watchdog.timer
systemctl list-timers service-watchdog.timer
sudo systemctl status service-watchdog.timer

The paired service executes the following command as root. It reads an optional /etc/service-watchdog/environment file and interprets Watchdog exit code 1 as an expected successful service result. Exit code 2 is consequently visible as a failed unit and should be investigated. 2

/opt/service-watchdog/service-watchdog.sh -c /etc/service-watchdog/config.yaml

Inspect a run with both the journal and Watchdog's configured operational log.

sudo journalctl -u service-watchdog.service -n 50 --no-pager
sudo tail -n 50 /var/log/service-watchdog/service-watchdog.log

Change the cadence

The installed timer contains OnUnitActiveSec=1min. Change the interval only after considering the worst-case duration of checks, retries, remediation commands, and post-remediation verification. Watchdog uses a global non-blocking lock so a concurrent run is skipped safely, but consistently skipped runs indicate that the schedule is too frequent or command timeouts are too large. 1 3

A site-specific override or a locally managed copy of the unit is preferable to editing a file that may be replaced during an upgrade. After changing unit files, reload systemd and restart the timer.

sudo systemctl daemon-reload
sudo systemctl restart service-watchdog.timer
systemctl list-timers service-watchdog.timer

Packaged unit property	Operational implication
Type=oneshot	Each timer event starts one complete Watchdog run.
User=root, Group=root	Docker or systemctl remediation can work without an extra privilege layer; restrict the configuration accordingly.
EnvironmentFile=-/etc/service-watchdog/environment	The leading - means the environment file is optional.
SuccessExitStatus=1	A service outage or attempted remediation is not treated as a systemd unit failure.
UMask=0027	Files created by the service have restrictive default permissions.
NoNewPrivileges, PrivateTmp, ProtectHome, ProtectSystem=full	Basic hardening is applied to the packaged service. 2

Use cron

Use root's crontab if the watchdog needs permission to call Docker, systemctl, or another privileged remediation command. Before adding the entry, make the executable available, create the log directory, and run a dry validation. 3

sudo chmod +x /opt/service-watchdog/service-watchdog.sh
sudo install -d -m 0750 /var/log/service-watchdog

sudo /opt/service-watchdog/service-watchdog.sh \
  -c /etc/service-watchdog/config.yaml \
  -n

sudo crontab -e

The following entry runs Watchdog every minute. Set an explicit PATH; cron has a deliberately limited environment. 3

PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

* * * * * /opt/service-watchdog/service-watchdog.sh -c /etc/service-watchdog/config.yaml >> /var/log/service-watchdog/cron.log 2>&1

For a five-minute interval, use:

*/5 * * * * /opt/service-watchdog/service-watchdog.sh -c /etc/service-watchdog/config.yaml >> /var/log/service-watchdog/cron.log 2>&1

When SMTP credentials are provided via an environment variable, supply them through a protected cron environment or another secret mechanism. Never put secrets in a world-readable crontab or configuration file. 3

Verify cron execution

sudo systemctl status cron
sudo crontab -l
sudo tail -f /var/log/service-watchdog/service-watchdog.log

Create /var/log/service-watchdog before installing the crontab entry. Shell redirection opens cron.log before Watchdog runs, so Watchdog cannot create a missing parent directory for that redirection. 3

Scheduling checklist

Check	Why it matters
Dry run passed under the same account used by the scheduler.	Confirms configuration, permissions, working directories, and basic connectivity.
Scheduler interval exceeds normal worst-case runtime.	Reduces skipped runs and misleading gaps in monitoring.
Command timeouts are explicit and bounded.	Limits lock duration and prevents hanging remediation.
Log and state paths are writable only as needed.	Protects diagnostics, state transitions, and cooldown markers.
Scheduler environment contains necessary secret variables.	Enables SMTP and hook integrations without embedding credentials in YAML.
A controlled failure and recovery were observed.	Confirms remediation and transition alerts before a production incident.

For correct handling of 1 and 2 in monitoring dashboards or job wrappers, see Remediation, State, and Exit Codes.

References