Skip to content

Retention Replay and Recovery

Jump to bottom
s9terpsync-release-bot edited this page Aug 5, 2026 · 1 revision

Retention, Replay & Recovery

This page is for whoever cleans up after an interrupted or partially-failed run: pruning old state with retention, figuring out what an incomplete run actually left behind with replay, and recovering SFTP files that got stuck mid-acquisition with sftp recover-claims.

It doesn't cover the run itself — that's Running S9TerpSync, which also documents run resume and the --retry-token / --retry-manifest / --retry-all flags this page's retry manifests feed into. It doesn't cover the config fields behind retention: — that's Configuration Reference, which documents the four policy blocks (runs, reports, referenceCache, archives) this page's commands act on.

Why replay needs manual help at all

S9TerpSync doesn't retain the raw bytes of a source record after processing it — there's no cache of "the exact payload Slate sent" to replay from later. That's a deliberate storage/privacy tradeoff, but it means a record that failed or was left in doubt can't be automatically re-driven from stored state. Every replay command's output reflects this directly: replay inspect always reports its plan as status="AUTOMATIC_REPLAY_UNAVAILABLE" reason="RAW_PAYLOAD_NOT_RETAINED", and replay assess-records always reports status="IDENTICAL_SOURCE_REINGESTION_REQUIRED" reason="RAW_PAYLOAD_NOT_RETAINED". Recovery instead means: figure out which records are safe to retry (assess-records), hand that list to a manifest (retry-manifests create), and reprocess them by pointing a normal run resume at the identical source data again (--retry-manifest).

retention

Two subcommands, both reading the same four policy blocks from s9terpsync.yaml: runs, reports, referenceCache, and archives. Each policy has an action of retain, archive, or delete — see Configuration Reference for the field rules (archive/delete requires afterDays; archives itself can't use archive). Only entries whose configured action is archive or delete and are older than that policy's afterDays ever show up in a plan — retain classes never produce plan entries.

retention preview

s9terpsync retention preview --file <path>

Fully read-only: computes the retention plan against the current state directory and prints it, without acquiring the run lock, writing anything, or requiring --dry-run/--live (there's no mode flag on preview at all).

Output — one retention candidate line per planned entry, one retention issue line per class the planner couldn't scan, then a summary line:

retention candidate className="runs" action="delete" relativePath="runs/2026-01-15T02-00-00Z.json" cutoff="2026-06-01T00:00:00.000Z" observed="2025-12-01T00:00:00.000Z"
retention issue className="reports" code="CLASS_SCAN_FAILED"
retention preview planned=1 archived=0 deleted=0 skipped=0 failed=0 issues=1

archived/deleted/skipped/failed are always 0 for preview — nothing was executed, only planned. Fails with retention preview failed, exit 1, if the plan itself can't be computed (for example an unreadable state directory).

Example:

s9terpsync retention preview --file ./s9terpsync.yaml

retention run

s9terpsync retention run --file <path> (--dry-run | --live)

Executes the same plan preview would show. Like run and run resume on Running S9TerpSync, retention run acquires the shared run lock in both --dry-run and --live mode — none of the three exempts --dry-run from lock contention. run recover-lock (also on Running S9TerpSync) is the actual exception: it uses a separate lock-recovery mechanism entirely and never touches the shared run lock, in either mode, which is why it can't fail with run lock already held. retention run itself will contend with an in-progress run, run resume, retention run, or sftp recover-claims, and fails with run lock already held, exit 3, if it can't get the lock.

  • --file (required) — path to s9terpsync.yaml.
  • --dry-run / --live — exactly one is required.
    • DRY_RUN plans and reports what would happen but performs no archive/delete: every planned entry is counted as skipped (not archived or deleted), and failed is always 0.
    • LIVE actually archives or deletes each planned entry, incrementing archived, deleted, or failed per entry as it goes.

Same candidate/issue/summary output shape as preview, with the mode word changed:

retention dry-run planned=1 archived=0 deleted=0 skipped=1 failed=0 issues=0

retention live planned=1 archived=0 deleted=1 skipped=0 failed=0 issues=0

In --live mode, if any entry failed to execute or the plan carried any retention issue, the command exits 2 (the summary line still prints first). --dry-run never exits 2 for this reason, even if the plan has issues — only --live treats a bad plan as a failure. Bad arguments (neither or both of --dry-run/--live) fail immediately with retention run mode: specify exactly one of --dry-run or --live, exit 1.

Example:

s9terpsync retention run --file ./s9terpsync.yaml --dry-run

replay

The replay command group is read-only investigation and manifest management — it never mutates a run's records directly. Reprocessing only happens through run resume (see Running S9TerpSync).

replay list-incomplete

s9terpsync replay list-incomplete --file <path>

Scans the state directory for runs that aren't in a terminal state yet, plus any runs the inventory couldn't safely classify at all.

Output:

replay list-incomplete status="OK" scanned=12 nonTerminal=2 terminal=10 blocked=0
replay incomplete runId="2026-08-01T02-00-00Z" mode="LIVE" status="NON_TERMINAL" files=3 records=140 succeeded=120 warned=5 failed=10 held=5

One replay incomplete line per non-terminal run. status on the summary line is "OK" when every scanned run was safely classified (having incomplete runs listed is normal and not itself an error), or "BLOCKED" when one or more runs' state couldn't be read/classified safely — that's an integrity signal, not just "work to do." The command exits 2 when blocked > 0; otherwise it exits 0 regardless of nonTerminal. Exits 1 with replay list-incomplete failed if the scan itself errors.

Example:

s9terpsync replay list-incomplete --file ./s9terpsync.yaml

replay inspect

s9terpsync replay inspect --file <path> --run-id <id>

Shows one run's full replay state — file and record counts, operation counts, and the target-evidence breakdown that determines what can be retried.

  • --file (required) — path to s9terpsync.yaml.
  • --run-id (required) — the run to inspect (matches the runId printed by run, run resume, or replay list-incomplete).

Output:

replay inspect runId="2026-08-01T02-00-00Z" status="COMPLETED_WITH_ERRORS" terminal=true mode="LIVE" files=3 records=140 succeeded=120 warned=5 failed=10 held=5 skipped=0 operations=140 operationFailures=10 idempotentOperations=140
replay evidence total=15 notAttempted=0 alreadySucceeded=0 retryableOnIdenticalSourceReingestion=8 blockedInDoubt=2 blockedFinalFailure=3 insufficientOrStaleEvidence=2
replay plan status="AUTOMATIC_REPLAY_UNAVAILABLE" reason="RAW_PAYLOAD_NOT_RETAINED"
replay warning code="RECORD_FAILURES_PRESENT"

Zero or more trailing replay warning lines can appear, with code one of RUN_NOT_TERMINAL, RECORD_FAILURES_PRESENT, or TARGET_OPERATION_FAILURES_PRESENT. Exits 0 on success regardless of what the run contains — inspect is purely informational; fails with replay inspect failed, exit 1, only if the run can't be read at all (unknown --run-id, corrupt state, etc.).

Example:

s9terpsync replay inspect --file ./s9terpsync.yaml --run-id 2026-08-01T02-00-00Z

replay assess-records

s9terpsync replay assess-records --file <path> --run-id <id> [--emit-retry-tokens]

The step before creating a retry manifest: classifies every failed/held record in a run into exactly one retry-eligibility bucket.

  • --file (required) — path to s9terpsync.yaml.
  • --run-id (required) — the run to assess.
  • --emit-retry-tokens (optional) — additionally print one opaque per-record retry token for each record classified RETRYABLE_ON_IDENTICAL_SOURCE_REINGESTION. These are the same tokens run resume --retry-token accepts for retrying individual records without a manifest.

Output without --emit-retry-tokens:

replay assess-records status="COMPLETED_WITH_ERRORS" terminal=true mode="LIVE" records=140 failed=10 held=5 skipped=0 candidates=15
replay record-assessment retryableOnIdenticalSourceReingestion=8 blockedInDoubt=2 blockedFinalFailure=3 insufficientOrStaleEvidence=2 notAttempted=0 notAssessed=0
replay plan status="IDENTICAL_SOURCE_REINGESTION_REQUIRED" reason="RAW_PAYLOAD_NOT_RETAINED"

With --emit-retry-tokens, one extra line per retryable record is inserted before the replay plan line:

replay record-retry token="sha256:9f2c1a4b7e60d3218ffa5b7c9e0d1a2b6c4e8f1a0d2b3c5e7f9a1b3c5d7e9f10" classification="RETRYABLE_ON_IDENTICAL_SOURCE_REINGESTION"

candidates always equals failed + held, and the six assessment buckets (retryableOnIdenticalSourceReingestion, blockedInDoubt, blockedFinalFailure, insufficientOrStaleEvidence, notAttempted, notAssessed) always sum to candidates. The command exits 2 — "blocked" — when the run isn't terminal yet, or when any record fell into blockedInDoubt, blockedFinalFailure, insufficientOrStaleEvidence, or notAssessed; it exits 0 only when every candidate record cleanly resolved to either already-succeeded or retryableOnIdenticalSourceReingestion. Exits 1 with replay assess-records failed if the assessment itself can't be computed.

Example:

s9terpsync replay assess-records --file ./s9terpsync.yaml --run-id 2026-08-01T02-00-00Z --emit-retry-tokens

replay retry-manifests

A retry manifest is a durable, named set of retryable records from one run, with a TTL — the thing run resume --retry-manifest <id> consumes instead of a long list of --retry-token flags. Four subcommands cover its lifecycle: create, list, revoke, cleanup.

create

s9terpsync replay retry-manifests create --file <path> --run-id <id> --ttl-days <days> (--dry-run | --live)

Re-runs the same record assessment assess-records would (internally, with retry tokens always collected) and, in --live mode, writes a manifest listing every record found RETRYABLE_ON_IDENTICAL_SOURCE_REINGESTION. You don't need to have run assess-records --emit-retry-tokens first — create does its own assessment.

  • --file, --run-id (required).
  • --ttl-days <days> (required) — a positive integer from 1 to 31; how many days from now the manifest stays valid before it's a cleanup candidate.
  • --dry-run / --live — exactly one is required.

Output:

replay retry-manifests create mode=LIVE created=1 manifestId="retry-manifest-20260801T020000Z-9f2c1a4b7e60" sourceRunId="2026-08-01T02-00-00Z" status="ACTIVE" expiresAt="2026-08-08T02:00:00.000Z" records=8

--dry-run prints wouldWrite=1 created=0 and omits manifestId (nothing was written); --live prints created=1 with the assigned manifestId. Fails with replay retry-manifests failed, exit 1, if there are no retryable records to put in a manifest, if the run/config can't be read, or on any other input problem.

Example:

s9terpsync replay retry-manifests create --file ./s9terpsync.yaml --run-id 2026-08-01T02-00-00Z --ttl-days 7 --live

list

s9terpsync replay retry-manifests list --file <path> [--run-id <id>]

Lists manifests, optionally filtered to one source run. No --dry-run/--live — this is read-only.

replay retry-manifests list count=2
replay retry-manifests list manifestId="retry-manifest-20260801T020000Z-9f2c1a4b7e60" sourceRunId="2026-08-01T02-00-00Z" status="ACTIVE" records=8
replay retry-manifests list manifestId="retry-manifest-20260728T110000Z-1a2b3c4d5e6f" sourceRunId="2026-07-28T11-00-00Z" status="REVOKED" records=3

Example:

s9terpsync replay retry-manifests list --file ./s9terpsync.yaml --run-id 2026-08-01T02-00-00Z

revoke

s9terpsync replay retry-manifests revoke --file <path> --manifest-id <id> (--dry-run | --live)

Marks a manifest REVOKED so it can no longer be used with run resume --retry-manifest. --dry-run reports the manifest's current record count without changing anything (fails if the manifest doesn't exist); --live actually revokes it.

replay retry-manifests revoke mode=LIVE revoked=1 records=8

Example:

s9terpsync replay retry-manifests revoke --file ./s9terpsync.yaml --manifest-id retry-manifest-20260728T110000Z-1a2b3c4d5e6f --live

cleanup

s9terpsync replay retry-manifests cleanup --file <path> --older-than-days <days> (--dry-run | --live)

Deletes manifests that are either expired or revoked, and have been so for at least --older-than-days days.

  • --older-than-days <days> (required) — a positive integer from 1 to 31, same bounds as --ttl-days on create.
  • --dry-run / --live — exactly one is required. DRY_RUN reports candidates without deleting; LIVE deletes them.
replay retry-manifests cleanup mode=LIVE candidates=3 deleted=3 skipped=0

Example:

s9terpsync replay retry-manifests cleanup --file ./s9terpsync.yaml --older-than-days 14 --dry-run

All four retry-manifests subcommands share one error path: any failure — bad input, missing manifest, unreadable store — is reported as replay retry-manifests failed, exit 1.

sftp recover-claims

This command lives under sftp, not replay — it's about recovering stuck source files, not run records.

s9terpsync sftp recover-claims --file <path> --older-than-minutes <minutes> (--dry-run | --live)

Before acquiring a file from the SFTP source, S9TerpSync renames it from the inbound path into a claim path, so two concurrent runs can't pick up the same file. If a run crashes or is killed after claiming a file but before finishing with it, that file is stranded in the claim path and won't be picked up by a normal run. sftp recover-claims finds claimed files older than --older-than-minutes and renames them back to the inbound path so a subsequent run can acquire them again.

  • --file (required) — path to s9terpsync.yaml. Requires sources.sftp.enabled: true in that config; otherwise fails with sources.sftp.enabled: configuration requirement not met.
  • --older-than-minutes <minutes> (required) — a positive integer; only claims older than this are eligible.
  • --dry-run / --live — exactly one is required, same rule as elsewhere on this page.

Like retention run, this command acquires the shared run lock in both modes, and fails with run lock already held, exit 3, if it's contended.

Output:

sftp claim recovery DRY_RUN scanned=4 eligible=2 recovered=0 skipped=2 failed=0

sftp claim recovery LIVE scanned=4 eligible=2 recovered=2 skipped=0 failed=0

In --live mode, if any recovery attempt failed, the command exits 2. Bad mode arguments fail with sftp recover-claims mode: specify exactly one of --dry-run or --live, exit 1.

Example:

s9terpsync sftp recover-claims --file ./s9terpsync.yaml --older-than-minutes 30 --live

The replay/retry lifecycle

Putting the replay commands and run resume together, a typical recovery looks like this:

flowchart TB
  runDone["Run completes<br/>COMPLETED_WITH_ERRORS · records failed or held"]
  listIncomplete["replay list-incomplete<br/>find the run among non-terminal runs"]
  inspect["replay inspect<br/>see per-run file/record/evidence detail"]
  assess["replay assess-records<br/>classify each failed/held record"]
  gate{"Any records<br/>RETRYABLE_ON_IDENTICAL_SOURCE_REINGESTION?"}
  blocked["Blocked records remain<br/>blockedInDoubt · blockedFinalFailure · insufficientOrStaleEvidence<br/>exit 2 — needs manual investigation, not auto-retryable"]
  manifest["replay retry-manifests create<br/>writes an ACTIVE manifest of retryable records"]
  resume["run resume --retry-manifest<br/>reprocesses only the manifest's records"]
  outcome["Resumed run reports<br/>COMPLETED or COMPLETED_WITH_ERRORS"]

  runDone --> listIncomplete
  listIncomplete --> inspect
  inspect --> assess
  assess --> gate
  gate -->|no retryable records| blocked
  gate -->|yes| manifest
  manifest --> resume
  resume --> outcome
  outcome -->|still COMPLETED_WITH_ERRORS| listIncomplete

  classDef pipeline fill:#EEEDFE,stroke:#534AB7,color:#26215C
  classDef mutation fill:#FAECE7,stroke:#993C1D,color:#4A1B0C
  classDef failure fill:#FAEEDA,stroke:#854F0B,color:#412402
  classDef readonly fill:#E1F5EE,stroke:#0F6E56,color:#04342C
  classDef neutral fill:#F1EFE8,stroke:#5F5E5A,color:#2C2C2A
  class runDone,gate neutral
  class listIncomplete,inspect,assess readonly
  class blocked failure
  class manifest,resume mutation
  class outcome pipeline
Loading

run resume --retry-manifest <manifest-id> is documented in full on Running S9TerpSync, including its mutual exclusivity with --retry-token and --retry-all.

Where to go next

Clone this wiki locally