Operations and Troubleshooting

Reliable operation starts with observability. Watchdog writes structured, line-oriented operational messages to settings.log_file; the packaged configuration uses /var/log/service-watchdog/service-watchdog.log. When the packaged systemd unit is used, its standard service output is also available through the journal. 1 2

Inspect a run

Use the commands below to inspect the most recent activity. Log records include a timestamp, level, service name where relevant, action/result labels, and bounded diagnostic detail. Command output is sanitized and truncated before it is written to the operational log. 1

sudo tail -n 100 /var/log/service-watchdog/service-watchdog.log
sudo journalctl -u service-watchdog.service -n 100 --no-pager

# Follow new activity during a controlled test.
sudo tail -f /var/log/service-watchdog/service-watchdog.log

Log pattern	Meaning	Next action
result=check-success	A check attempt succeeded.	No action unless the result conflicts with expected service behavior.
result=check-failed	One attempt failed; retries may remain.	Review the detail value and network/service logs.
result=unavailable	All attempts failed and the service remained unavailable.	Check remediation status, cooldown, and target diagnostics.
action=remediation-start	A remediation sequence is starting.	Confirm commands, privileges, working directory, and timeout.
result=recovered-after-remediation	Verification succeeded after remediation.	Review the root cause; Watchdog's process exit is still 1 because remediation was attempted.
result=email-sent / result=email-failed	SMTP notification delivery outcome.	Investigate any failed delivery promptly.
action=lock result=already-running	Another Watchdog process holds the global lock.	Review scheduler interval and worst-case command duration. 1

Inspect state and cooldowns

The configured state_directory stores two kinds of per-service files: <service>.state contains healthy or unavailable, and <service>.last-action contains the epoch timestamp of the most recent remediation attempt. Watchdog treats an absent or invalid state file as unknown. 1

sudo ls -la /var/lib/service-watchdog
sudo cat /var/lib/service-watchdog/public-api.state
sudo cat /var/lib/service-watchdog/public-api.last-action

Do not edit these files during a running invocation. If a controlled test needs a fresh initial state, stop the scheduler first and remove only the target service's state files. The next failure will be considered a new transition and may therefore emit a failure notification. See Persistent state and transition behavior for the transition table.

Common problems

Symptom	Likely cause	Resolution
yq: command not found or an unsupported yq error	A missing binary or the unrelated Python yq package.	Install Mike Farah yq v4 and confirm yq --version reports v4.
Configuration error before checks start	Invalid YAML, missing required fields, duplicate/invalid service name, bad path, or invalid command-array shape.	Run service-watchdog.sh -c CONFIG -n and correct the exact validation message.
working_directory does not exist	A command item refers to a directory that is missing on this host.	Create the directory or correct working_directory; it must exist during validation.
HTTP result is unexpectedly unhealthy	Endpoint URL, redirect behavior, method, expected status, certificate, DNS, or timeout is wrong.	Test with the same curl semantics; review success_status, follow_redirects, and timeout.
TCP result is unavailable	Listener not present, wrong host/port, firewall, or service is overloaded.	Verify the service listener locally and network reachability from the Watchdog host.
Remediation reports permission denied	The scheduler account cannot use Docker, systemctl, or the configured executable.	Use the least-privileged account that has the necessary access, or use the packaged root unit with a protected configuration.
Remediation is skipped for cooldown	A previous remediation timestamp remains within actions.cooldown.	Wait, adjust the cooldown deliberately, or reset the relevant .last-action during a controlled maintenance window.
A scheduled run is skipped	A prior run holds the global lock.	Check the log, command timeouts, and scheduler interval; avoid simply disabling the lock.
Cron does not create cron.log	The redirect target's parent directory did not exist before cron ran.	Create /var/log/service-watchdog before installing the crontab entry.
No test email appears	All services remained healthy, so no state transition occurred.	Create a controlled failure/recovery scenario; do not expect mail from a healthy first run. 1 3

Safe diagnostic workflow

First reproduce the behavior with a targeted dry run. This confirms YAML validation and executes only the check, without remediation, hooks, mail, or state writes. Then inspect the target service directly using the same account and environment as the scheduler. 1

# Targeted, non-destructive check.
sudo /opt/service-watchdog/service-watchdog.sh \
  -c /etc/service-watchdog/config.yaml \
  -s public-api \
  -n

# Confirm the exact version in use.
sudo /opt/service-watchdog/service-watchdog.sh --version

If the dry run succeeds but scheduled runs fail, compare the scheduler's user, PATH, environment variables, service hardening, and filesystem permissions. In particular, an SMTP password_env must be available to the systemd service or cron process that invokes Watchdog. 2 4

Security and permissions

Operate Watchdog with the least privilege needed by its remediation actions. Keep the YAML configuration administrator-owned and not writable by the service account; do not embed shell snippets, tokens, or passwords in it. The program directly executes YAML command arrays rather than using eval or configured bash -c, but a user who can modify the configuration can still change what privileged Watchdog executes. 1 3

Control	Recommended practice
Configuration ownership	Use root:root and restrictive permissions such as 0640; grant a group only when a non-root operator genuinely needs read access.
Secrets	Prefer notifications.email.smtp.password_env; for the packaged unit, use /etc/service-watchdog/environment with mode 0600.
Remediation	Permit only commands and working directories that have been reviewed for the target host.
Privilege	Run as root only when an action truly requires it; otherwise use a dedicated account with narrowly scoped access.
Logs and state	Keep their directories non-world-writable; review them as part of incident response.
Certificates	Keep tls_required: true and insecure_skip_verify: false for normal Internet-facing SMTP. 1 3

Project test suite

Before submitting changes to the project or adopting a modified version, run the repository's syntax checks, ShellCheck, smoke test, and notification test. The smoke test verifies healthy and remediation paths using a temporary local HTTP server; the email test verifies transition-only behavior. 5

bash -n service-watchdog.sh install.sh tests/smoke.sh
bash -n tests/email-notifications.sh
shellcheck service-watchdog.sh install.sh tests/smoke.sh tests/email-notifications.sh
bash ./tests/smoke.sh
bash ./tests/email-notifications.sh

Escalation checklist

When reporting an issue, include the Watchdog version, sanitized configuration fragment, exact invocation command, relevant operational-log lines, scheduler type, and the expected versus observed behavior. Never include SMTP passwords, access tokens, or other credentials.

References