Skip to content

Disaster Recovery Ragnarok Drill

Jump to bottom
Frank Yglesias Bertheau edited this page Jul 7, 2026 · 1 revision
<h1>ᚾ&nbsp;NornGate — Disaster Recovery (Ragnarök Drill)</h1>

Rehearse the end so the rebirth is clean. In the myth the world burns and is reborn from the few who survive; the drill proves NornGate's survivors — the signed checkpoint, the immutable ledger, the config versions, the key — are enough to rebuild from. Ragnarök is rehearsed, not discovered.

RPO/RTO figures are targets from the recovery model; the drill ragnarok command is documented. Reconcile specifics against your deployment.

ᚨ Before you start

The recovery model is four moves: the signed checkpoint is the seed → restore to it → replay forward → verify → resume.

  • Targets: RPO 5 minutes (checkpoint/snapshot cadence), RTO 15 minutes (restore + replay + verify).
  • Staging first. A drill is a controlled failure — scope it with --target=staging before you ever run it against production.
  • Cadence. Run it on a schedule. Recovery you have not rehearsed is a hypothesis, not a capability.

ᛈ What survives the fall

Recovery is only as good as what is preserved off the failing path. Four survivors, each in a different trust domain so one failure cannot take all of them:

  • The signed checkpoint — the current chain-head, the seed the world is rebuilt from.
  • The cold ledger in Niflheim — append-only, Object Lock immutable, 7-year retention; the full history to replay.
  • The config versions (gatecfg_*, pol_*) — so the reborn fleet evaluates identically, not approximately.
  • The signing key in the Asgard HSM — a separate trust domain from the workers; without it you can neither verify the chain nor re-sign.

Why the separation matters: if the checkpoint or the key lived on the workers that burned, there would be nothing to rebuild from. The trust-domain split is precisely what makes rebirth possible — the survivors have to survive the same fire that took the fleet.

ᛉ Degraded mode during the outage

Before recovery completes, the fleet fails closed but gracefully: read-only and pre-authorized actions continue on cached policy/state snapshots, while anything needing a fresh G1/G2 decision denies.

Why: availability is graded by risk class — safe reads keep serving, but no gate opens on an unverified write. Fail-closed does not mean fully dark; it means the dark parts are exactly the risky ones (Security & Fail-closed).

ᚺ Run the drill

norngate-cli drill ragnarok --target staging --duration 30m
norngate-cli drill status

The drill injects a controlled failure (kills the primary ledger or region), forces the recovery path, measures RPO/RTO, and reports. Why a --duration: the window holds the failure long enough to exercise degraded-mode behavior, not just the restore blip — you are testing what users experience during the outage as much as the recovery itself.

ᛒ The recovery sequence

  1. Restore from the latest signed checkpoint — establishes a trusted chain-head.
  2. Replay forward from the Niflheim cold ledger past the checkpoint to the last durable record — rebuilds state deterministically.
  3. Verify the chain — norngate-cli audit verify walks from the checkpoint, checking hashes and Ed25519 signatures. A break aborts recovery; you do not resume on a corrupt or tampered ledger.
  4. Reprocess the dead-letter — work that failed mid-flight is re-driven from Hel by Hermóðr; idempotency keys ensure re-driven effects commit once.
  5. Resume — gates return to full evaluation; degraded mode lifts.

Why replay is safe at all: determinism plus idempotency. Replayed decisions re-derive identically (same pinned inputs → same verdict), and replayed effects do not duplicate (exactly-once at G4). Without those two properties, replay would be a second, divergent run — with them, it is reconstruction (Deterministic Execution).

ᛞ Verify

Five pass criteria separate "we restarted" from "we recovered, provably":

  • RPO — data loss within the checkpoint window (≤ target).
  • RTO — time to resume (≤ target).
  • Chain — audit verify clean over the recovery window.
  • No double-commit — the outcome count for replayed requests is unchanged; idempotency held.
  • Replay match — a sampled replayed decision equals its original verdict; determinism held.

Why all five: a drill that only checks "the service came back" proves nothing about correctness. The chain check proves integrity, no-double-commit proves the replay did not duplicate side effects, and the replay-match proves the reborn fleet decides the same way the old one did.

ᚱ Next steps

Iconography

Section glyphs are Elder Futhark runes (Unicode Runic block, U+16A0–U+16FF) — semantic, not emoji. Full set on Home:

Rune Name Gloss Marks
ᚾ Nauðiz need, constraint the platform mark
ᚨ Ansuz the word, instruction before you start
ᛈ Perthro the well of Urd, what persists what survives the fall
ᛉ Algiz protection, warding degraded mode
ᚺ Hagalaz hail, Ragnarök run the drill
ᛒ Berkanan growth, rebirth the recovery sequence
ᛞ Dagaz dawn, the new day verify
ᚱ Raidō the ride, the road next steps

References. RFC 6962 (signed log, checkpoint verification) + Ed25519 / RFC 8032; S3 Object Lock (immutable cold storage); Kafka replay; RPO/RTO (disaster-recovery practice); determinism + exactly-once, per Deterministic Execution. Eddic source: Völuspá (Ragnarök and the rebirth), per Norse Cosmology & Platform Design.

ᚾ NornGate — Disaster Recovery (Ragnarök Drill)

Rehearse the end so the rebirth is clean. In the myth the world burns and is reborn from the few who survive; the drill proves NornGate's survivors — the signed checkpoint, the immutable ledger, the config versions, the key — are enough to rebuild from. Ragnarök is rehearsed, not discovered.

RPO/RTO figures are targets from the recovery model; the drill ragnarok command is documented. Reconcile specifics against your deployment.

ᚨ Before you start

The recovery model is four moves: the signed checkpoint is the seed → restore to it → replay forward → verify → resume.

  • Targets: RPO 5 minutes (checkpoint/snapshot cadence), RTO 15 minutes (restore + replay + verify).
  • Staging first. A drill is a controlled failure — scope it with --target=staging before you ever run it against production.
  • Cadence. Run it on a schedule. Recovery you have not rehearsed is a hypothesis, not a capability.

ᛈ What survives the fall

Recovery is only as good as what is preserved off the failing path. Four survivors, each in a different trust domain so one failure cannot take all of them:

  • The signed checkpoint — the current chain-head, the seed the world is rebuilt from.
  • The cold ledger in Niflheim — append-only, Object Lock immutable, 7-year retention; the full history to replay.
  • The config versions (gatecfg_*, pol_*) — so the reborn fleet evaluates identically, not approximately.
  • The signing key in the Asgard HSM — a separate trust domain from the workers; without it you can neither verify the chain nor re-sign.

Why the separation matters: if the checkpoint or the key lived on the workers that burned, there would be nothing to rebuild from. The trust-domain split is precisely what makes rebirth possible — the survivors have to survive the same fire that took the fleet.

ᛉ Degraded mode during the outage

Before recovery completes, the fleet fails closed but gracefully: read-only and pre-authorized actions continue on cached policy/state snapshots, while anything needing a fresh G1/G2 decision denies.

Why: availability is graded by risk class — safe reads keep serving, but no gate opens on an unverified write. Fail-closed does not mean fully dark; it means the dark parts are exactly the risky ones ([Security & Fail-closed](Security-and-Fail-Closed)).

ᚺ Run the drill

norngate-cli drill ragnarok --target staging --duration 30m
norngate-cli drill status

The drill injects a controlled failure (kills the primary ledger or region), forces the recovery path, measures RPO/RTO, and reports. Why a --duration: the window holds the failure long enough to exercise degraded-mode behavior, not just the restore blip — you are testing what users experience during the outage as much as the recovery itself.

ᛒ The recovery sequence

  1. Restore from the latest signed checkpoint — establishes a trusted chain-head.
  2. Replay forward from the Niflheim cold ledger past the checkpoint to the last durable record — rebuilds state deterministically.
  3. Verify the chain — norngate-cli audit verify walks from the checkpoint, checking hashes and Ed25519 signatures. A break aborts recovery; you do not resume on a corrupt or tampered ledger.
  4. Reprocess the dead-letter — work that failed mid-flight is re-driven from Hel by Hermóðr; idempotency keys ensure re-driven effects commit once.
  5. Resume — gates return to full evaluation; degraded mode lifts.

Why replay is safe at all: determinism plus idempotency. Replayed decisions re-derive identically (same pinned inputs → same verdict), and replayed effects do not duplicate (exactly-once at G4). Without those two properties, replay would be a second, divergent run — with them, it is reconstruction ([Deterministic Execution](Deterministic-Execution)).

ᛞ Verify

Five pass criteria separate "we restarted" from "we recovered, provably":

  • RPO — data loss within the checkpoint window (≤ target).
  • RTO — time to resume (≤ target).
  • Chain — audit verify clean over the recovery window.
  • No double-commit — the outcome count for replayed requests is unchanged; idempotency held.
  • Replay match — a sampled replayed decision equals its original verdict; determinism held.

Why all five: a drill that only checks "the service came back" proves nothing about correctness. The chain check proves integrity, no-double-commit proves the replay did not duplicate side effects, and the replay-match proves the reborn fleet decides the same way the old one did.

ᚱ Next steps

Iconography

Section glyphs are Elder Futhark runes (Unicode Runic block, U+16A0–U+16FF) — semantic, not emoji. Full set on [Home](Home#iconography):

Rune Name Gloss Marks
ᚾ Nauðiz need, constraint the platform mark
ᚨ Ansuz the word, instruction before you start
ᛈ Perthro the well of Urd, what persists what survives the fall
ᛉ Algiz protection, warding degraded mode
ᚺ Hagalaz hail, Ragnarök run the drill
ᛒ Berkanan growth, rebirth the recovery sequence
ᛞ Dagaz dawn, the new day verify
ᚱ Raidō the ride, the road next steps

References. [RFC 6962](https://www.rfc-editor.org/rfc/rfc6962) (signed log, checkpoint verification) + [Ed25519 / RFC 8032](https://www.rfc-editor.org/rfc/rfc8032); S3 Object Lock (immutable cold storage); Kafka replay; RPO/RTO (disaster-recovery practice); determinism + exactly-once, per [Deterministic Execution](Deterministic-Execution). Eddic source: Völuspá (Ragnarök and the rebirth), per [Norse Cosmology & Platform Design](Norse-Cosmology-and-Platform-Design).

Welcome

Getting Started

Core Concepts

Guides

Reference

Silence means no.

Clone this wiki locally