Skip to content

db: verify the tail of the chain on startup and back out of a bad tip - #61

Merged
Bitflash-sh merged 1 commit into
mainfrom
verify-chain-on-load
Jul 31, 2026
Merged

db: verify the tail of the chain on startup and back out of a bad tip#61
Bitflash-sh merged 1 commit into
mainfrom
verify-chain-on-load

Conversation

@Bitflash-sh

Copy link
Copy Markdown
Owner

A node that accepted blocks its current rules would reject — because it was running an older build, or because the tip it landed on was never valid — had no way to notice. It kept the branch, kept building on it, and kept mining coins the rest of the network throws away. The only cure was deleting blk*.dat by hand, which requires knowing something is wrong in the first place.

A node here once mined a 323-block fork for five days exactly like that.

Satoshi added this upstream after the block 74638 incident, for the same reason: so nobody would have to delete files.

What it does

On load, walk back from the tip running CheckBlock on each block. If one fails, move the best chain back to its parent, disconnect what came after, and let the node re-download from peers.

Only the last 288 blocks are checked by default — about ten hours at a two-minute target — because every check costs a RandomX hash. /checkblocks=N changes it, 0 does the whole chain.

Two things worth reviewing

The repair runs in LoadBlockIndex() in main.cpp, not in CTxDB::LoadBlockIndex where the detection happens. CTxDB::LoadBlockIndex walks the index with a cursor it never closes, so it is still holding read locks when it returns, and a write transaction opened alongside it waits on them forever. That is measured, not guessed: the first version came up, printed the diagnosis, and hung there with nothing in any log. Closing that cursor is its own fix — it is one of the things Satoshi fixed in c5c7911da that this fork never got — and is deliberately not bundled here.

A failed rollback reports and starts anyway. Refusing to start would leave the operator with no node to read the message from.

What it does not catch

What CheckBlock checks: proof of work, merkle root, duplicate transactions, sigops, timestamps. It does not re-verify signatures or spends — that would mean rebuilding the transaction index. So a tip that is invalid only under the script rules from #15 is not caught by this.

Testing

Forcing a block at tip-5 to fail verification:

LoadBlockIndex(): hashBestChain=55b9f6a291a9fe  height=3922
LoadBlockIndex() : *** bad block at height 3917, hash=3767c2cb9dc601
LoadBlockIndex(): verified 288 block(s)
LoadBlockIndex() : *** moving best chain back to height 3916, discarding 6 block(s)
LoadBlockIndex() : now at height 3916, will re-download from peers

The node stayed up and re-synced past its old tip, to 3928. Without the forced failure it verifies 288 and says so. Every error in the log during the run was a dead .btf descriptor or the down Nostr relay — none from this path.

A node that accepted blocks its current rules would reject -- because it
was running an older build, or because the tip it landed on was never valid
-- had no way to notice. It kept the branch, kept building on it, and kept
mining coins the rest of the network throws away. The only cure was
deleting blk*.dat by hand, which requires knowing something is wrong. A
node here once mined a 323-block fork for five days that way.

On load, walk back from the tip running CheckBlock on each block. If one
fails, move the best chain back to its parent, disconnect what came after,
and let the node re-download from peers. Satoshi added this after the block
74638 incident for the same reason: so it would not have needed anyone to
delete files.

Only the last 288 blocks are checked by default -- about ten hours at a
two-minute target -- because every check costs a RandomX hash.
/checkblocks=N changes it, 0 does the whole chain.

The repair deliberately runs in LoadBlockIndex() in main.cpp rather than in
CTxDB::LoadBlockIndex where the detection happens. CTxDB::LoadBlockIndex
walks the index with a cursor it never closes, so it still holds read locks
when it returns, and a write transaction opened alongside it waits on them
forever. That is measured, not guessed: the first version came up, printed
the diagnosis, and hung with nothing in any log. Closing that cursor is a
separate fix worth making on its own.

A failed rollback reports and starts anyway. Refusing to start would leave
the operator with no node to read the message from.

What this catches is what CheckBlock catches: proof of work, merkle root,
duplicate transactions, sigops, timestamps. It does not re-verify
signatures or spends, which would mean rebuilding the transaction index.

Verified by forcing a block at tip-5 to fail: the node reported the bad
block, moved back six, stayed up, and re-synced past its old tip. Without
the forced failure it verifies 288 and says so.
@Bitflash-sh
Bitflash-sh merged commit d975f61 into main Jul 31, 2026
@Bitflash-sh
Bitflash-sh deleted the verify-chain-on-load branch July 31, 2026 11:53
@Bitflash-sh Bitflash-sh mentioned this pull request Jul 31, 2026
Bitflash-sh added a commit that referenced this pull request Jul 31, 2026
Six changes since 1.2.7, and the reason not to sit on this is #58:
coinbase transactions were not unique, two of them overwrote older ones
on the main chain, and 100 BTF stopped being reachable. Nothing in how
they were built had changed since, so it could happen again at any time.
The height now goes into the coinbase on both paths that build one
(#59, #60), which makes a repeat impossible without any coordination.

Also in: a node now notices a peer it can no longer hear and stays
audible to peers that check the same way (#55); it announces its height
in the handshake and says out loud when it falls behind the network
(#56); script evaluation has ceilings on operation count and stack depth
(#57); the tail of the chain is verified at startup and a bad tip is
backed out of instead of being mined on (#61); and the wallet now
believes the chain about what it has already spent, which is what a
restored backup gets wrong (#62).
Bitflash-sh added a commit that referenced this pull request Jul 31, 2026
)

Three functions open a cursor and none of them close it: CTxDB::LoadBlockIndex,
CTxDB::ReadOwnerTxes and CWalletDB::LoadWallet.

Berkeley DB requires every cursor to be closed before the handle it came
from. ~CDB() calls Close(), which calls pdb->close() inside a catch-all
that swallows whatever it returns -- so the violation was real and silent,
which is this codebase's favourite kind.

An open cursor also holds read locks, and that is not theoretical here: the
first version of the startup chain repair (#61) opened a write transaction
beside the cursor LoadBlockIndex was still holding and waited on it
forever, coming up, printing its diagnosis and hanging with nothing in any
log. The repair had to be moved to the caller to get clear of it.

A scope guard rather than a close call at each return, because all three
functions have several exits and the point is to stop relying on anyone
remembering. Same fix Bitcoin made in c5c7911da.

Verified: node starts, verifies 288 blocks, loads the wallet, runs the
spent-flag rescan and syncs the live chain to 4422 with no Berkeley DB
error anywhere and db.log still empty.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant