Skip to content

Diagnostics and Troubleshooting

Jump to bottom
s9terpsync-release-bot edited this page Aug 10, 2026 · 2 revisions

Diagnostics & Troubleshooting

This page is for whoever gets paged when a run fails, or wants to check a configuration and environment before flipping anything to --live/LIVE. It covers the three read-only diagnostic commands — the top-level diagnose, and ethos contracts status / ethos contracts live-target-readiness — where to look when a run or recovery command fails, and the exit-code conventions used consistently across the whole CLI.

It doesn't cover fixing a failed run's records — that's Retention, Replay & Recovery, which documents replay list-incomplete, replay inspect, replay assess-records, and retry manifests in full. It doesn't cover the config fields these commands check — that's Configuration Reference.

diagnose

s9terpsync diagnose --file <path>

diagnose is a top-level command — s9terpsync diagnose, not s9terpsync config diagnose. It's entirely local and read-only: it never contacts Slate, Ethos, or Banner. It loads the config, then reports:

  • which sources are enabled (sources.sftp.enabled / sources.slateApi.enabled),

  • schedule posture (schedule.enabled, schedule.mode, schedule.source),

  • processing status (whether reference-data/processing is enabled),

  • live-target readiness, computed from the same local Ethos contract evidence ethos contracts live-target-readiness reads (no live Ethos call is made — a DISABLED value here just means the check wasn't applicable, not that it was skipped over a network problem), and

  • a run-assessment scan of <stateDirectory>/runs — how many stored runs are non-terminal, terminal, or couldn't be classified at all.

  • --file (required) — path to s9terpsync.yaml.

Output — three lines:

diagnose scope="LOCAL_ONLY" status="CLEAR" sources="SFTP"
diagnose schedule status="DISABLED" mode="DRY_RUN" source="AUTO" processing="DISABLED" liveTarget="DISABLED"
diagnose run-assessment scanned=0 nonTerminal=0 terminal=0 blocked=0

status is one of CLEAR, ADVISORY, or BLOCKED:

  • BLOCKED — liveTarget="NOT_READY", or the run assessment found one or more runs it couldn't safely classify (blocked > 0). Exits 2.
  • ADVISORY — otherwise fine, but one or more non-terminal runs are sitting in <stateDirectory>/runs (nonTerminal > 0) — worth a look with replay list-incomplete, not itself a failure. Exits 0.
  • CLEAR — none of the above. Exits 0.

If the config can't be loaded, or the local evidence/run state is unreadable or inconsistent, it fails with local diagnostics failed, exit 1, rather than reporting a status at all.

Example:

s9terpsync diagnose --file ./s9terpsync.yaml

ethos contracts status

s9terpsync ethos contracts status

This lives under ethos contracts — s9terpsync ethos contracts status, not a bare s9terpsync ethos status. It takes no options and doesn't need --file: it reports the contract-evidence manifests bundled with the installed package (contracts/fixtures/ethos), one line per known contract:

Ethos contract readiness
contract=ethos/persons/create resource=persons version=12 status=vendor-openapi-evidenced capability=mutation liveReadEnabled=false liveMutationEnabled=false remainingGates=6
contract=ethos/persons/current-person resource=persons version=12 status=s9texp-verified capability=read liveReadEnabled=true liveMutationEnabled=false remainingGates=1

status is one of fixture-only, s9texp-verified, vendor-openapi-evidenced, or institution-verified — how far that contract's evidence has progressed, not whether it's safe to write live yet (that's what remainingGates, and live-target-readiness below, are for). This is purely informational: it exits 0 as long as the evidence manifests can be read at all, regardless of any individual contract's status, and fails with ethos contracts status failed, exit 1, only if the manifests themselves can't be read.

Example:

s9terpsync ethos contracts status

ethos contracts live-target-readiness [--root <path>]

s9terpsync ethos contracts live-target-readiness [--root <path>]

Also nested under ethos contracts. This rolls the same per-contract evidence status reads up into a single aggregate verdict for whether the full set of contracts S9TerpSync needs is ready for LIVE writes:

  • --root <path> (optional) — where to read contract-evidence manifests from, in place of the default packaged contracts/fixtures/ethos directory. Useful for pointing the check at institution-specific evidence you've generated separately.
Ethos live target contract readiness status=NOT_READY requiredContracts=20 issues=56
contract=ethos/persons/create reason=NOT_INSTITUTION_VERIFIED
contract=ethos/persons/create reason=LIVE_MUTATION_DISABLED
contract=ethos/persons/create reason=REMAINING_GATES

status is READY only when issues=0; otherwise it's NOT_READY, with one contract=... reason=... line per outstanding gap (reasons include things like NOT_INSTITUTION_VERIFIED, LIVE_READ_DISABLED, LIVE_MUTATION_DISABLED, REMAINING_GATES, and MISSING).

This command's exit code doesn't reflect readiness — like ethos contracts status, it exits 0 whenever it successfully renders a result, even when status=NOT_READY. It fails with ethos contracts live-target-readiness failed, exit 1, only if the evidence manifests can't be read. You have to read the status= field yourself; don't rely on the exit code to gate a --live cutover.

Example:

s9terpsync ethos contracts live-target-readiness --root ./contracts/fixtures/ethos

Where to look when a run fails

flowchart TB
  start["Something looks wrong"]
  diagnose["diagnose --file &lt;path&gt;<br/>local, read-only, no network calls"]
  status{"diagnose status"}
  blockedLive["liveTarget=NOT_READY"]
  readiness["ethos contracts live-target-readiness<br/>which contracts, which reason"]
  blockedRuns["run-assessment blocked&gt;0"]
  advisory["run-assessment nonTerminal&gt;0"]
  incomplete["replay list-incomplete<br/>find the non-terminal/blocked runs"]
  runReport["Check the run's own report<br/>&lt;stateDirectory&gt;/reports, path printed as<br/>'report path=...' by run/run resume"]
  runManifest["Check the run's manifest<br/>&lt;stateDirectory&gt;/runs, read via<br/>replay inspect --run-id &lt;id&gt;"]

  start --> diagnose
  diagnose --> status
  status -->|BLOCKED, liveTarget| blockedLive --> readiness
  status -->|BLOCKED, run assessment| blockedRuns --> incomplete
  status -->|ADVISORY| advisory --> incomplete
  status -->|CLEAR but a specific run failed| runReport
  incomplete --> runManifest
  runManifest --> runReport

  classDef pipeline fill:#EEEDFE,stroke:#534AB7,color:#26215C
  classDef mutation fill:#FAECE7,stroke:#993C1D,color:#4A1B0C
  classDef failure fill:#FAEEDA,stroke:#854F0B,color:#412402
  classDef readonly fill:#E1F5EE,stroke:#0F6E56,color:#04342C
  classDef neutral fill:#F1EFE8,stroke:#5F5E5A,color:#2C2C2A
  class start,status neutral
  class diagnose,readiness,incomplete,runManifest readonly
  class blockedLive,blockedRuns,advisory failure
  class runReport pipeline
Loading

Two places hold the actual evidence, both under the config's stateDirectory:

  • <stateDirectory>/reports — every run and run resume writes a report here and prints its location on the report path=<relativePath> checksum=<checksum> sizeBytes=<n> line (see Running S9TerpSync). Each per-record issue in that report carries a category (VALIDATION, MATCHING, CROSSWALK, BUSINESS_RULE, EXTERNAL_API, or SYSTEM) and a code — that's the fastest way to tell a bad crosswalk from a bad match from an Ethos-side rejection.
  • <stateDirectory>/runs — the durable run manifest for every run, the thing diagnose's run-assessment scans and replay list-incomplete / replay inspect --run-id <id> read. Use those commands rather than reading the JSON directly; see Retention, Replay & Recovery for the full replay toolset and the retry-manifests lifecycle that follows from it.

When the report and the manifest both look unremarkable but the run still went wrong, turn the log stream up: LOG_LEVEL=debug s9terpsync run --file <path> --source sftp --dry-run re-runs it with the per-stage and per-record debug lines included. Those log lines are structured JSON on stdout, one object per line — but they aren't the command's only output: the human-readable result lines go to stdout too, and diagnostics and any uncaught stack trace go to stderr, none of it JSON. So pipe the combined stream through a filter that tolerates non-JSON lines rather than a bare jq, which aborts on the first one:

LOG_LEVEL=debug s9terpsync run --file <path> --source sftp --dry-run 2>&1 | jq -R 'fromjson? // empty'

-R feeds each line in as a raw string, fromjson? parses the JSON ones and emits nothing for the rest, and // empty drops any line that parses to a bare null or false. Drop the 2>&1 (and read stderr separately) when you want the plain-text output too. For the daemon, the same variable is set as a systemctl edit drop-in and needs a restart to take effect; see Service & Systemd Operation, whose "Reading the log stream" section covers the same filter against journalctl. Note that an invalid level fails the CLI with exit 1 rather than falling back to the default — the valid values are fatal, error, warn, info, debug, trace, and silent.

Exit codes across the CLI

The same four exit codes mean the same thing everywhere they're used:

Exit code Meaning Where it applies
0 Success. Includes diagnose status="CLEAR"/"ADVISORY", and ethos contracts status/live-target-readiness rendering successfully regardless of what they report. All commands.
1 A usage/mode error, or a runner-level failure that isn't specifically a blocked/failed-live outcome — e.g. neither or both of --dry-run/--live, an unreadable or invalid --file, an unmet source precondition. All commands that can fail.
2 A blocked or failed-live outcome: diagnose status="BLOCKED"; run/run resume reporting COMPLETED_WITH_ERRORS; a --live retention run, sftp recover-claims, service recover-lock, or run recover-lock reporting any failed entry; replay list-incomplete status="BLOCKED"; replay assess-records finding any non-retryable/unassessed record. diagnose, run, run resume, run recover-lock, retention run, sftp recover-claims, service recover-lock, replay list-incomplete, replay assess-records.
3 The shared run.lock is already held by another run — printed as run lock already held. run, run resume, retention run, sftp recover-claims, service recover-lock. Note run recover-lock itself never exits 3 this way — it's the command that inspects/clears that lock, not one that contends for it.

config validate and config show follow the same 0/1 split as everything else (schema/validation problems and unreadable files exit 1), but they never write to stateDirectory and so never touch exit codes 2 or 3.

Common failure signals

Signal Likely cause What to check
config validate --file <path> prints one or more <field>: <problem> lines (for example environment: unsupported value) and exits 1 The YAML doesn't satisfy the configuration schema — a missing required field, wrong type, or an invalid cross-field combination Fix each reported field against Configuration Reference, then rerun config validate until it prints configuration valid for <institutionId>
diagnose reports status="BLOCKED" with liveTarget="NOT_READY" One or more required Ethos contracts aren't ready for live use yet ethos contracts live-target-readiness for the per-contract reason= values, and the ethos.resourceVersions / ethos.apiKeySecretRef fields behind them
run, run resume, retention run, sftp recover-claims, or service recover-lock fails immediately with run lock already held (exit 3) The shared run.lock is genuinely held by an in-progress run, or was left stranded by a crashed process If nothing is actually running, recover the stale lock — run recover-lock for the CLI's per-run lock, service recover-lock for the daemon's service-level lock (see Running S9TerpSync and Service & Systemd Operation)
run/run resume reports status COMPLETED_WITH_ERRORS (exit 2) Some records in the run failed or were held Open the run's report under <stateDirectory>/reports (path from the report path=... line) for per-record category/code, then replay inspect --run-id <id> and replay assess-records to see what's retryable
A run report issue has category="CROSSWALK" (for example code UNMAPPED_REQUIRED_CROSSWALK or MISSING_REQUIRED_CROSSWALK_SOURCE) An incoming source value has no matching entry in the crosswalk registry Add or enable the missing entry in the config's crosswalks: section (domain / sourceValue / targetCode) — see Configuration Reference
sftp recover-claims fails with sources.sftp.enabled: configuration requirement not met The command was pointed at a config where the SFTP source isn't enabled Confirm sources.sftp.enabled: true in the --file you're passing; this command only applies to the SFTP source
replay list-incomplete reports status="BLOCKED" (exit 2) One or more runs under <stateDirectory>/runs couldn't be safely classified — a state-integrity signal, not ordinary backlog Investigate the affected run manifests directly rather than treating it like a normal non-terminal count; see Retention, Replay & Recovery
Scheduled runs aren't behaving as expected (not firing, or firing in the wrong mode) schedule.enabled / schedule.mode misconfigured — note there is no execution.mode field; dry-run vs. live for manual commands is chosen per-invocation with --dry-run/--live, and for the schedule specifically it's schedule.mode Check schedule.enabled, schedule.mode (DRY_RUN or LIVE, required whenever enabled: true), and schedule.source in s9terpsync.yaml — see Configuration Reference

Where to go next

  • Running S9TerpSync — run, run resume, and run recover-lock, and how --dry-run/--live work across them.
  • Retention, Replay & Recovery — the full replay toolset (list-incomplete, inspect, assess-records, retry-manifests) for turning a COMPLETED_WITH_ERRORS run into a clean retry.
  • Service & Systemd Operation — service recover-lock and the service-level lock, for a daemon that won't start.
  • Configuration Reference — every field referenced in the troubleshooting table above, in full.

Clone this wiki locally