Skip to content

Service and Systemd Operation

Jump to bottom
s9terpsync-release-bot edited this page Aug 10, 2026 · 2 revisions

Service & Systemd Operation

This page is for whoever stands S9TerpSync up as a long-lived, unattended service instead of running it by hand. It covers the two service subcommands (service plan and service recover-lock), the on-disk directory layout the systemd unit expects, and the service-level lock that keeps two daemon instances from running against the same state at once.

It doesn't cover installing the software in the first place — that's Installation & Requirements, which also documents the s9terpsync-service binary itself. It doesn't cover writing s9terpsync.yaml — that's Configuration Reference. And it doesn't cover manual, one-off runs — that's Running S9TerpSync, which also documents run recover-lock, the CLI-driven counterpart to this page's service recover-lock.

There is no service run or service preview

The service command group has exactly two subcommands: plan and recover-lock. There is no service run, service start, or service preview — nothing under service starts the daemon or executes a sync.

The daemon itself, s9terpsync-service, is a separate binary (see Installation & Requirements). It's started and supervised by systemd via the packaged unit file, not invoked directly by an operator at a shell prompt, and it isn't a subcommand of the s9terpsync CLI at all.

service plan

s9terpsync service plan [--config <path>] [--package <name>]

Prints a deterministic, non-mutating RHEL/systemd installation checklist. Running it does not create a service account, write any file, create any directory, resolve any secret, contact Slate/Ethos/Banner, or enable/start systemd — it's guidance to follow by hand (or to feed into your own configuration-management tooling), not an installer.

  • --config <path> (optional) — the config file path shown throughout the printed checklist. Defaults to /etc/s9terpsync/s9terpsync.yaml. Display-only: passing a different path doesn't create or check anything at that path, it just changes what the checklist text says.
  • --package <name> (optional) — the npm package name shown in the install step. Defaults to @soft9tech/s9terpsync. Also display-only.

Both options are validated for safe characters (no control characters, no path traversal segments in --config, a well-formed package-name shape in --package); an unsafe value fails with service plan failed, exit 1.

The checklist itself walks eight stages: confirm the RHEL 8/9 + Node.js 24 runtime, install the npm package, create the dedicated service account and the four directories described below, generate and validate configuration with s9terpsync config init / config validate, install the packaged systemd unit and run systemctl daemon-reload, prove readiness with per-source --dry-run calls before touching anything live, deliberately edit the config to turn scheduling on, and only then systemctl enable --now the service.

Example:

s9terpsync service plan --config /etc/s9terpsync/s9terpsync.yaml

Directory layout

The plan and the packaged systemd unit agree on the same four paths:

Path Purpose
/etc/s9terpsync Holds s9terpsync.yaml. Referenced by the unit's S9TERPSYNC_CONFIG environment variable.
/etc/s9terpsync/secrets Holds the secret files referenced by the config's file:/absolute/path secret references (SFTP password, Slate API token, Ethos API key).
/var/lib/s9terpsync The config's stateDirectory. All file-based durable state lives here: run manifests, locks (including the service-level lock below), indexes, reports, archives, quarantine, and retention audit records. There's no database — this directory is it.
/var/log/s9terpsync Reserved, writable application-log space — expect it to stay empty. The application writes no log files at all today; its log stream goes to stdout only, which under the unit flows into journald (journalctl -u s9terpsync, see Reading the log stream below). The directory exists and is granted write access ahead of any future file-based logging.

The example configuration at config/s9terpsync.example.yaml sets stateDirectory: /var/lib/s9terpsync to match.

The systemd unit

The packaged unit template ships at assets/systemd/s9terpsync.service:

[Unit]
Description=S9TerpSync Sync Engine
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=s9terpsync
Group=s9terpsync
Environment=S9TERPSYNC_CONFIG=/etc/s9terpsync/s9terpsync.yaml
ExecStart=/usr/bin/env s9terpsync-service
Restart=on-failure
RestartSec=10
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/var/lib/s9terpsync /var/log/s9terpsync
UMask=0027

[Install]
WantedBy=multi-user.target

A few things worth calling out:

  • Environment=S9TERPSYNC_CONFIG=... is how the daemon finds its config file. s9terpsync-service reads the S9TERPSYNC_CONFIG environment variable (falling back to /etc/s9terpsync/s9terpsync.yaml if unset) — there's no --file flag for the daemon the way there is for the CLI's run/service recover-lock commands.
  • ExecStart=/usr/bin/env s9terpsync-service runs the daemon binary by name off PATH, which is what a global npm install gives you.
  • User=s9terpsync / Group=s9terpsync — the dedicated non-login service account created in service plan step 3, not the operator's own account.
  • ProtectSystem=strict, ProtectHome=true, ReadWritePaths=/var/lib/s9terpsync /var/log/s9terpsync — the filesystem is read-only everywhere except those two paths (plus PrivateTmp's isolated /tmp). The daemon can't write to /etc/s9terpsync at runtime; configuration and secrets are operator-managed, not self-modified.
  • Restart=on-failure / RestartSec=10 — systemd restarts the daemon on crash, 10 seconds later, rather than leaving it down.
  • NoNewPrivileges=true / UMask=0027 — standard privilege-escalation and default-permission hardening for a service account.

Reading the log stream

The daemon's log output is structured JSON: one object per line, written to stdout, which the unit hands to journald. That is not the whole of what the unit emits, though. systemd captures stderr into the same journal, and a few things there are fixed human-readable text rather than JSON:

  • the startup diagnostic, printed on every start once the startup run assessment finishes — service startup run assessment status="CLEAR" scanned=<n> nonTerminal=<n> terminal=<n> blocked=<n>, or the bare line service startup run assessment failed when that assessment itself can't complete;
  • the stack trace of any error that escapes startup, including the invalid-LOG_LEVEL failure described below.

So the journal for the unit is mostly JSON, not entirely JSON. -o cat strips journald's own metadata; it does not separate the two streams or filter out the non-JSON lines, and a bare | jq pipe aborts on the first one it reaches. Read the stream back with a filter that skips them instead:

journalctl -u s9terpsync -o cat | jq -R 'fromjson? // empty'

-o cat still matters: without it, journalctl prefixes each line with its own timestamp, hostname, and unit name, and nothing after the prefix parses. -R then hands each line to the filter as a raw string rather than letting jq's own parser reject it, fromjson? parses the ones that are JSON and — via the trailing ? — yields nothing at all for the ones that aren't, and // empty does the same for a line that parses to a bare null or false. The usual filters compose onto the end of it: jq -R 'fromjson? // empty | select(.level >= 50)' for errors and worse, jq -R 'fromjson? // empty | select(.runId == "…")' to follow a single run.

The plain-text lines are still worth reading — they're the service's only human-readable startup signal — so reach for an unfiltered journalctl -u s9terpsync -o cat when you're diagnosing a start that didn't happen, and for the jq pipe when you're reading the log stream of one that did.

Setting the level. Verbosity comes from the LOG_LEVEL environment variable — fatal, error, warn, info, debug, trace, or silent, matched case-insensitively, defaulting to info when unset or empty. The packaged unit above deliberately doesn't set it, so changing it means a drop-in rather than editing the shipped file:

sudo systemctl edit s9terpsync

and, in the editor that opens:

[Service]
Environment=LOG_LEVEL=debug

Then restart the daemon:

sudo systemctl restart s9terpsync

The restart isn't optional. The level is read once, when the logger is constructed at startup, and is never re-read — a running daemon keeps the level it started with no matter what the drop-in now says. systemctl daemon-reload on its own isn't enough either: it reloads the unit definition, not the already-running process.

An invalid value is fatal, not ignored. A LOG_LEVEL that isn't one of those seven values doesn't fall back to info — the daemon refuses to start. It fails during startup, before it binds its HTTP port or acquires service.lock, and nothing catches the resulting error, so what reaches stderr (and so the journal) is an uncaught LogLevelError — a plain-text, multi-line Node stack trace whose first line reads LogLevelError: LOG_LEVEL must be one of: fatal, error, warn, info, debug, trace, silent. It is not a JSON log line, and the jq pipe above will drop it; use an unfiltered journalctl -u s9terpsync -o cat to see it. The rejected value is never echoed back. Because the unit sets Restart=on-failure with RestartSec=10, systemd then restarts it into exactly the same failure, every ten seconds, indefinitely — a typo here looks like a restart loop, not like a service running at the wrong verbosity.

To get out of that loop, correct the override you actually added — run sudo systemctl edit s9terpsync again and either fix the Environment=LOG_LEVEL= line or delete it, then sudo systemctl restart s9terpsync. Removing the line returns the daemon to the default info. Resist reaching for systemctl revert s9terpsync unless you mean it: revert discards every local customization of the unit — all drop-ins and any full unit-file override in /etc/systemd/system — not just the LOG_LEVEL line, so on a host where anything else has been customized it will silently undo that too.

The service-level lock

s9terpsync-service acquires a lock file at <stateDirectory>/locks/service.lock once, at startup, before it starts listening or activates the scheduler — and holds it for the entire lifetime of the process, releasing it only on graceful shutdown. Its only job is to stop two daemon instances from running against the same state directory at the same time. If a second instance tries to start while the lock is held, it fails fast rather than running alongside the first.

This is a different lock from run.lock, which is acquired per-run (manual or scheduled) and is covered on Running S9TerpSync. A running service still acquires run.lock around each individual sync it performs; service.lock is the outer, whole-process lock that exists for as long as the daemon does.

Because the lock is only released on a clean shutdown, an ungraceful stop — a SIGKILL, an OOM kill, a host power loss — can strand service.lock on disk with no process left holding it. When that happens, restarting the service fails until the stale lock is cleared. That's what service recover-lock is for.

service recover-lock

s9terpsync service recover-lock --file <path> --older-than-minutes <minutes> (--dry-run | --live)

Recovers a stale service-level lock so the daemon can start again.

  • --file <path> (required) — path to s9terpsync.yaml. As with the CLI's other commands, this points at the configuration file, not at the lock file itself; the config's stateDirectory is what tells the command where locks/service.lock actually lives.
  • --older-than-minutes <minutes> (required) — a positive integer. Only a lock held longer than this many minutes is eligible for recovery; zero, negative, or non-numeric values fail validation.
  • --dry-run / --live — exactly one is required. DRY_RUN reports whether the lock is eligible without touching it; LIVE actually recovers it.

The command inspects only <stateDirectory>/locks/service.lock. A missing lock is reported ABSENT (nothing to recover). A lock that's still within the age threshold, or whose owning process is still alive, is ACTIVE and left alone. A lock that's aged out and whose owning process is verifiably dead is ELIGIBLE. Anything ambiguous or unsafe — a malformed or unreadable lock file, a symlink where a regular file is expected, a future-dated timestamp, a permission error — is BLOCKED and left alone rather than guessed at.

LIVE recovery of an eligible lock momentarily acquires run.lock (to avoid racing an in-progress run), quarantines the stale lock file under quarantine/stale-locks/, and appends a value-safe audit record to retention/service-lock-recovery-audit.jsonl before reporting RECOVERED.

On success it prints:

service lock recovery <mode> status=<status> scanned=<n> eligible=<n> recovered=<n> skipped=<n> failed=<n>

Exit codes: 1 if the mode flags or --older-than-minutes fail validation; 3 if run.lock is already held by something else (printed as run lock already held); 2 in LIVE mode if the recovery attempt itself failed; 0 otherwise.

Example:

s9terpsync service recover-lock --file /etc/s9terpsync/s9terpsync.yaml --older-than-minutes 30 --dry-run

Where to go next

Clone this wiki locally