State Recovery

State Recovery

The daemon owns its state. Operators own their recovery.

The daemon owns its state. Operators own their recovery.

The one rule, restated: do not edit state.json, state_meta.json, state.db, claim files, or transcripts in place. The atomic-write and lock discipline only holds for the programmatic API. The moment you hand-edit a state file, the crash-safety guarantees stop applying — and they stop applying to you.

There are two state backends, and recovery differs between them. New installs use JSON (state_backend: json, the default). If you ran caduceus migrate-state --to-sqlite, you're on SQLite. Pick the section that matches your state_backend.

How you find out it's corrupt

• caduceus status --json — JSON backend: the state_corrupt field flips to true. SQLite backend: the top-level diagnostic field reports corrupt_state or corrupt_queue (the state_corrupt field is a JSON-backend signal and stays false for SQLite).
• The daemon's stderr — it exits with a StateCorrupt error and a non-zero code.
• A marker file appearing: state.json.corrupt (JSON backend). The daemon refuses to call the GitHub API while a marker is present.

The daemon never silently truncates or overwrites a corrupt file. It preserves the original and archives it with a timestamp before anything else happens.

The JSON recovery path

1. Stop the daemon. Kill the active tick however you started it (Hermes cron, system cron, manual). New ticks would race your recovery.
2. Read the marker. cat $STATE_DIR/state.json.corrupt — the file is empty; its presence is the signal.
3. Read the archive. The original lives at state.json.corrupt-<unix-ts> (or the metadata equivalent). Open it; understand what's wrong. The usual causes: a crash mid-write (file a bug — the atomic-write discipline should prevent this), an operator hand-edit (you know who you are), or filesystem corruption (check dmesg).
4. Build a repaired file. It must parse as the queue schema (entries as a map of display keys to records) and the metadata schema. If hand-writing that sounds terrible, it is — use caduceus migrate-state --from <prior-state.json> to let the daemon build it for you.
5. Apply it. The safe path: caduceus::migrate::recover_state (the library API, daemon-lock-protected — archives the corrupt original, atomically installs the repaired file, then clears the marker). The direct path: move the corrupt file aside, write the repaired content with the canonical temp + fsync + rename pattern, remove the marker. The library path is better; the direct path bypasses the daemon-lock protection.
6. Verify. caduceus status --json reports the recovered state and state_corrupt: false. If it doesn't, the recovery didn't take — don't push on.
7. Restart the daemon.

The SQLite recovery path

SQLite corruption surfaces as an open-time error rather than a marker file: the daemon detects it via PRAGMA integrity_check, a schema-version mismatch, or a structurally malformed database.

# confirm + scope the damage
sqlite3 $STATE_DIR/state.db "PRAGMA integrity_check;"        # "ok" = healthy
sqlite3 $STATE_DIR/state.db "SELECT MAX(version) FROM schema_version;"

1. Stop the daemon.
2. Back up first. sqlite3 $STATE_DIR/state.db ".backup $STATE_DIR/state.db.backup-$(date +%s)" — or VACUUM INTO, which rejects corrupt pages outright (a failure there means structural damage).
3. Restore or repair. Best: restore from a known-good backup via caduceus::migrate::recover_sqlite_state. Alternative: rebuild from a .dump. Surgical sqlite3 deletes for localised corruption are possible but advanced — and they bypass the daemon-lock protection.
4. Verify. The daemon should open the store cleanly — no corrupt_state/corrupt_queue diagnostic, no StateCorrupt error.
5. Restart.

If you have no backup and choose to create a fresh store, the queue state is gone — re-enqueue needed issues by re-adding their trigger labels.

The other recovery commands

• caduceus queue reset owner/repo#number — moves a Failed or Skipped entry back to Queued. The persisted finalization checkpoint (branch / PR / run / commit) is preserved so the next tick resumes from where the work actually got to. --force-finalization-reset drops the checkpoint — the daemon warns you and names the branch and PR; it never deletes them. --dry-run shows what would change. Removing and re-adding the trigger label is not a substitute: the worker-failure budget survives label churn; only the explicit reset clears it.
• caduceus queue reprocess owner/repo#number — new generation for an issue: increments the counter and moves a terminal entry back to Queued.
• caduceus migrate-state --to-sqlite — the backend switch. Imports JSON state into the SQLite store and flips state_backend in your config. --dry-run first, always.

Housekeeping that's yours

• Corrupt archives (state.db.corrupt-*) and manual backups (state.db.backup-*) aren't swept by the daemon. Keep the last few, delete the rest: ls -t $STATE_DIR/state.db.corrupt-* | tail -n +6 | xargs rm -f
• WAL files (state.db-wal, state.db-shm) are normal; SQLite checkpoints them. Never delete them by hand.

When to file a bug

• A corrupt archive that parses cleanly (the daemon's loader had a false positive — attach the archive).
• A refused recovery that was valid in your judgement (attach the corrupt original and your repaired file).
• Recovery succeeded but the next tick misbehaved (attach the recovered state and the tick log).

File at the project's GitHub issues. Don't include secrets — the daemon redacts tokens from its logs, but a state file you paste manually is on you.