perf(drive-abci): don't create GroveDB checkpoints while replaying history - #4553
perf(drive-abci): don't create GroveDB checkpoints while replaying history#4553PastaPastaPasta wants to merge 1 commit into
Conversation
…story Protocol version 11 turns checkpoints on, with a policy of one every 600 seconds of chain time keeping the last 3. Mainnet blocks are about 2.5 minutes apart, so that is a checkpoint every four blocks — which at replay speed is roughly 37 a second, each one a RocksDB checkpoint over the whole database plus a copy of the platform state, and all but the last three deleted again immediately. Measured at 15.1 ms a block from height 318,704, against about 7 ms for everything else in a block put together. A finished mainnet sync had four checkpoint directories left on disk out of some 26,000 created. Checkpoints are restore points for a running node, so skip them for blocks more than ten minutes old and write the first real one on reaching the tip. They live outside the tree, so no app hash changes.
|
Warning Review limit reachedNext included review available in 53 minutes. View limit detailsLimit details: You’ve used the included review currently available. You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. Review configuration: ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (3)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
🕓 Ready for review — 14 ahead in queue (commit 302f7ef) |
|
Superseded by #4570, which is the same change from a branch in 🤖 Posted autonomously by Claude on behalf of pasta. |
2 participants
Issue being fixed or feature implemented
A node replaying mainnet history creates and destroys tens of thousands of GroveDB checkpoints.
Protocol version 11 (mainnet height 318,704) turns checkpoints on:
with
frequency_seconds: 600, num_checkpoints: 3. The interval is in chain time, and mainnet blocks are ~2.5 minutes apart, so that is a checkpoint every four blocks. At replay speed that is roughly 37 a second, each one a RocksDB checkpoint over the whole database — hard-linking every SST, flushing and copying the WAL — plus a copy of the platform state, and all but the last three deleted again immediately.Measured with per-block phase timing, replaying mainnet:
against roughly 7,000 µs for everything else in a block put together. A finished mainnet sync left four checkpoint directories on disk out of some 26,000 created:
The last three are four blocks apart, which is the cadence.
It shows up as an I/O stall rather than CPU: during the affected range drive-abci sat at 0.38 of one core with the system 47–73% idle and the disk at 2,000–4,000 tps.
What was done?
Skip checkpoint creation for blocks more than ten minutes old. Checkpoints are restore points for a running node; a node catching up has no use for restore points into blocks it is about to replace, and it writes its first real checkpoint on reaching the tip.
The age test is a new
utils::is_historical_block, with unit tests — the predicate is the testable part, since a historical replay makes every block historical by construction.Checkpoints live outside the tree, so no app hash changes.
How Has This Been Tested?
Same 3,000-block window from mainnet height 331,648, two local peers, back to back:
fb_checkpoint−60.8% per block. The 47.6 blocks/s reproduces what a full sync does over that range.
Full mainnet replay, genesis to 424,981, with this change on top of an otherwise identical build: 4,684.9 s → 3,032.9 s, 35% of the whole sync. Every committed app hash matched a reference sync across all 424,971 heights.
cargo test -p drive-abci --lib should_checkpoint utils::replay— 11 passed.One existing test needed updating:
test_first_block_should_always_checkpointbuilt its context withblock_time_ms = 1_000_000, which is 1970 and now reads as history. It uses a current timestamp, and a companion test covers the new behaviour for an old block.Breaking Changes
A node interrupted mid-initial-sync will have no recent checkpoint. That is the case where a checkpoint is least useful — the remedy for trouble there is the resync it was already doing.
Checklist:
For repository code-owners and collaborators only
🤖 Generated with Claude Code