Skip to content

Recovery Incident Runbook

Alex Stoyanov edited this page Mar 4, 2026 · 2 revisions

Recovery Incident Runbook

Use this page during active recovery incidents when speed and consistency matter.

1) Stabilize First

  • Work with one backup set only.
  • Copy inputs to a working directory; keep originals read-only.
  • Record operator, timestamp, and incident ticket/context.
  • Confirm tool version: ethernity --version.

2) Pick the Recovery Path

Condition Path
Clean scan exports available --scan
Scan quality poor but fallback text available --fallback-file
Recovery requires shard/auth inputs main path + --shard-* and/or --auth-*

Codec reminders during incidents:

  • Scan inputs can carry raw frame bytes or base64 text; --scan handles both.
  • Payload text files (--payloads-file, --shard-payloads-file, --auth-payloads-file) must be canonical unpadded base64.
  • Envelope payload decoding follows manifest payload_codec (raw/gzip) after decryption.

3) Run Recovery

Primary scan path:

ethernity recover --scan ./incident/scans --output ./incident/recovered.bin

Fallback + shard/auth path:

ethernity recover \
  --fallback-file ./incident/recovery_fallback.txt \
  --shard-fallback-file ./incident/shard_a.txt \
  --shard-fallback-file ./incident/shard_b.txt \
  --auth-payloads-file ./incident/auth_payloads.txt \
  --output ./incident/recovered.bin

4) Validate Output

  • Compare recovered bytes against trusted source where available.
  • Preserve command history and output logs.
  • Keep input artifacts and recovered outputs together for audit.

5) Rescue Mode (Last Resort)

Use --rescue-mode only when:

  • standard auth verification blocks recovery,
  • incident approval is documented,
  • and output will be independently validated.

Rescue mode example:

ethernity recover --scan ./incident/scans --rescue-mode --output ./incident/recovered.bin

After rescue-mode recovery:

  • mark the output as incident-recovered,
  • preserve evidence,
  • and schedule post-incident trust review.

6) Escalation Triggers

Escalate and pause if you observe:

  • repeated parser/conflict failures with known-good artifacts,
  • unexpected signer/auth mismatch,
  • unreproducible output differences,
  • or provenance verification failures for binaries.

Related

Clone this wiki locally