Persisted mem-table checkpoints for crash recovery (Node #261)

[adamantmm]

Summary

The ADAMANT node now supports persisted, rotating checkpoints of derived mem_* state. After a forced interruption that leaves memory mirrors inconsistent, startup can restore the latest verified checkpoint and replay only blocks after the checkpoint height — instead of rebuilding all memory tables from height 1.

This implements the design from #227, merged in #261 (feat/memtable-checkpoints → dev, head 84dc3613).

Checkpoints are a local recovery cache only. Blocks and deterministic replay remain the source of truth. If verification or replay fails, the node falls back to the existing full rebuild path (fail-closed).

---

Problem

Derived tables (mem_accounts, mem_round, delegate/multisig junction tables, and their unconfirmed mirrors) can become inconsistent if the process is killed while block, account, or round writes are in flight:

[WRN] loader Detected unapplied rounds in mem_round
[WRN] loader Recreating memory tables…
[inf] loader Rebuilding blockchain, current block height: 1…

Graceful shutdown (Ctrl+C / SIGTERM) is still the required operational path. Checkpoints reduce recovery time when a forced kill happens anyway.

---

What was implemented

Storage (SQL migration)

• Metadata table: mem_state_checkpoint_meta (slot, schemaVersion, height, blockId, round, nethash, createdAt, status, digest)
• Three rotating slot table sets: mem_ckpt_0..2_* for confirmed state:
  • mem_accounts
  • mem_accounts2delegates
  • mem_accounts2multisignatures
  • mem_round
• Unconfirmed junction tables (mem_accounts2u_*) are not checkpointed; they are rebuilt from confirmed state on restore.

Migration: sql/migrations/20260709120000_createMemStateCheckpoints.sql

Core logic

Module	Role
logic/memCheckpoint.js	Digest, slot rotation, create/restore, invariant checks, recoverable checkpoint selection
modules/memCheckpoints.js	Module wrapper, onBlockApplied, tryRecover, schema guard on blockchainReady
sql/memCheckpoints.js	SQL helpers, canonical digest queries, copy/clear statement arrays
modules/loader.js	Recovery before full rebuild; partial replay via load({ startOffset, skipTableReset })
modules/blocks/chain.js	Trigger checkpoint after persisted block at completed round boundary

When checkpoints are created

• Only at completed round boundaries (heights 101, 202, 303, … on a 101-delegate network).
• After the full applyBlock pipeline has persisted the block.
• At chain tip (not syncing): every completed round.
• During catch-up sync: every 100th round (SYNC_ROUND_INTERVAL = 100) so sync throughput is not reduced by a full mem_* copy on every boundary.

Non-blocking copy (MVCC pin)

Checkpoint creation uses a PostgreSQL REPEATABLE READ transaction:

1. The MVCC snapshot is frozen by the transaction's first statement.
2. onPinned releases the block-processing critical section (library.sequence) as soon as the writing metadata row is persisted.
3. Table copy + digest + markMetaComplete continue in the background against the frozen snapshot — later blocks are invisible to it.
4. Guards before marking complete:
   • mem_round must be settled at the checkpoint round
   • max block height referenced by copied accounts must not exceed checkpoint height

This avoids holding the forging/sync critical section for the entire copy (~350–400 ms on testnet-sized state in live tests).

Verification chain (fail-closed)

Before a checkpoint is accepted for recovery:

1. status === 'complete'
2. schemaVersion matches
3. nethash matches node config
4. Referenced block exists at the recorded height
5. SHA-256 digest over canonical row serialization matches stored digest
6. Slot table schemas match live mem_* (on startup; disables creation if mismatch)
7. Post-restore invariants: accounts at checkpoint block, settled mem_round, no orphaned block refs, delegates present

Recovery tries all complete slots newest-first (getCompleteDesc), so one corrupted newest slot does not force a full rebuild if an older valid slot exists.

Startup recovery flow

1. checkMemTables() detects inconsistency → reload()
2. If loading.snapshot / verification mode is active → skip checkpoints, full genesis rebuild
3. Otherwise memCheckpoints.tryRecover():
   • restore slot → live mem_*
   • reset unconfirmed (u_*) state from confirmed mirrors
   • seed lastBlock from checkpoint block
   • replay blocks from checkpoint.height + 1 to tip
4. If replay fails → discard checkpoint state, full rebuild from genesis (persisted blocks are never clipped)

Configuration

config.default.json:

"loading": {
  "memCheckpoints": {
    "enabled": true
  }
}

Disable with "enabled": false if needed. Default is enabled.

---

Safety / consensus notes

• No protocol change: block/transaction bytes, signatures, delegate order, rewards, fees, and activation gates are unchanged.
• Checkpoints are never consensus input.
• Tampered local checkpoint data cannot bypass block validation — replay still goes through normal applyBlock.
• Manual SQL edits to mem_* remain unsafe; there is no supported deterministic repair without replay or a trusted DB snapshot.

---

Storage impact (from issue investigation)

On current mainnet-sized mem_* footprint (~48 MB total across six base tables):

• ~48 MB per in-database slot copy
• Three slots ≈ 96–144 MB plus metadata
• Conservative operator budget: reserve ~1 GB headroom for growth and rotation overhead

---

Validation performed

Unit tests: test/unit/logic/memCheckpoint.js — 18/18 passing (lifecycle, digest rejection, rotating-slot fallback, invariant short-circuit).

Manual testnet (162.55.32.80, adamanttest, branch 84dc3613):

• Checkpoint created at round boundary (height 409959, slot 1)
• Corruption test: DELETE FROM mem_accounts → restore from checkpoint → replay ~10 blocks → Blockchain ready (~78 ms restore-to-ready in that run)
• mem_accounts count restored correctly

Local testnet (test/config.json, memCheckpoints.enabled: true):

• Checkpoint creation during sync (height 404000, slot 2)
• Corruption recovery: restore + replay ~6098 blocks in ~15.6 s → mem_accounts = 1175, Blockchain ready

---

Operator reminders

1. Prefer graceful shutdown — wait for cleanup logs before restart (Cleaned up successfully).
2. Forced kill can still corrupt mem_*; checkpoints shorten recovery but do not replace correct shutdown.
3. After upgrading from pre-2d7cc2ce builds, expect a one-time digest-format mismatch on old checkpoints → fail-closed full rebuild, then new-format checkpoints.
4. Cross-repo operator docs in Adamant-im/docs are planned as follow-up (#261 notes).

---

References

• Issue: #227 — Persisted mem-table checkpoints for crash recovery
• Pull request: #261 — Feat: Add persisted mem-table checkpoints for crash recovery (merged 2026-07-10)
• Key files on dev: logic/memCheckpoint.js, modules/memCheckpoints.js, modules/loader.js, sql/memCheckpoints.js

Feedback from node operators — especially mainnet copy latency at round boundaries and recovery times on larger chains — is welcome.