feat(flatkv): add snapshot, WAL catchup, and rollback support by blindchaser · Pull Request #2972 · sei-protocol/sei-chain

blindchaser · 2026-02-24T15:15:47Z

Describe your changes and provide context

Introduce a snapshot-based lifecycle for FlatKV so that restarts replay
only from the nearest PebbleDB checkpoint instead of the full WAL.

Key changes:

Snapshot management: immutable PebbleDB checkpoints created via
Checkpoint(), managed through a "current" symlink and atomic
directory operations. Configurable interval, retention, and
minimum time between snapshots.
Working directory: mutable clone of the baseline snapshot (hardlinks
for .sst files) so writes never mutate snapshot dirs.
WAL catchup: on open, replay changelog entries from the snapshot
version to the target version using O(1) arithmetic + O(log N)
binary search for offset resolution.
Rollback: rewind to the best snapshot <= target, truncate WAL,
replay to exact version, and prune future snapshots.
File lock: prevent concurrent access from multiple processes.
Migration: automatically move pre-snapshot flat layout into a
versioned snapshot directory on first open.
Auto WAL truncation: periodically discard WAL entries older than
the earliest retained snapshot.
Fix account LtHash baseline capture to use pre-batch state when
multiple ApplyChangeSets calls precede a single Commit.
Add legacyDB to flushAllDBs.
Mark Iterator/IteratorByPrefix as EXPERIMENTAL (unused in production).

Testing performed to validate your change

- Use getAccountValue (cache-aware) instead of getAccountValueFromDB for LtHash old-value reads, fixing incorrect deltas across multiple ApplyChangeSets calls before Commit - Clear pending write buffers in open() to prevent stale writes after LoadVersion - Fix misleading "always sync" comment on WAL writes (NoSync=true)

github-actions · 2026-02-24T15:16:48Z

The latest Buf updates on your PR. Results from workflow Buf / buf (pull_request).

Build	Format	Lint	Breaking	Updated (UTC)
`✅ passed`	`✅ passed`	`✅ passed`	`✅ passed`	Feb 27, 2026, 6:20 PM

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: bccc292045

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-02-24T15:23:50Z

sei-db/state_db/sc/flatkv/snapshot.go

+	if err := s.open(); err != nil {
+		return fmt.Errorf("open for rollback: %w", err)
+	}


Reclone baseline snapshot before rollback catchup

When Rollback updates current and immediately calls open, it does not invalidate working/SNAPSHOT_BASE, so after a restart (where working was already cloned from that same snapshot) createWorkingDir reuses a working DB that may already be at a higher version than targetVersion. In that case catchup skips all entries (entry.Version <= committedVersion), rollback fails with a version mismatch, and this can happen after the WAL has already been truncated, leaving rollback in a partially-mutated state. Force a fresh clone of working from the selected snapshot before opening during rollback (like LoadVersion(target>0) already does).

Useful? React with 👍 / 👎.

sei-db/state_db/sc/flatkv/snapshot.go

+	s.pruneSnapshots(dir, version)
+
+	success = true
+	s.lastSnapshotTime = time.Now()


codecov · 2026-02-24T15:55:03Z

Codecov Report

❌ Patch coverage is 63.18408% with 222 lines in your changes missing coverage. Please review.
✅ Project coverage is 58.16%. Comparing base (ff1fcea) to head (8ea2bfe).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
sei-db/state_db/sc/flatkv/snapshot.go	66.86%	59 Missing and 50 partials ⚠️
sei-db/state_db/sc/flatkv/store.go	57.60%	30 Missing and 23 partials ⚠️
sei-db/state_db/sc/flatkv/store_catchup.go	58.33%	23 Missing and 17 partials ⚠️
sei-db/state_db/sc/flatkv/store_write.go	37.50%	8 Missing and 7 partials ⚠️
sei-db/state_db/sc/flatkv/store_lifecycle.go	76.47%	2 Missing and 2 partials ⚠️
sei-db/state_db/sc/flatkv/store_read.go	50.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2972      +/-   ##
==========================================
+ Coverage   58.14%   58.16%   +0.01%     
==========================================
  Files        2111     2113       +2     
  Lines      173562   174071     +509     
==========================================
+ Hits       100924   101247     +323     
- Misses      63683    63780      +97     
- Partials     8955     9044      +89

Flag	Coverage Δ
sei-chain-pr	`65.85% <63.18%> (?)`
sei-db	`69.50% <ø> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
sei-db/db_engine/pebbledb/db.go	`93.50% <100.00%> (+0.35%)`	⬆️
sei-db/state_db/sc/flatkv/config.go	`100.00% <100.00%> (ø)`
sei-db/state_db/sc/flatkv/iterator.go	`39.65% <100.00%> (+1.63%)`	⬆️
sei-db/state_db/sc/flatkv/keys.go	`100.00% <ø> (+4.70%)`	⬆️
sei-db/state_db/sc/flatkv/store_meta.go	`67.56% <100.00%> (ø)`
sei-db/state_db/sc/flatkv/store_read.go	`58.27% <50.00%> (+1.51%)`	⬆️
sei-db/state_db/sc/flatkv/store_lifecycle.go	`52.38% <76.47%> (+3.99%)`	⬆️
sei-db/state_db/sc/flatkv/store_write.go	`68.75% <37.50%> (-3.13%)`	⬇️
sei-db/state_db/sc/flatkv/store_catchup.go	`58.33% <58.33%> (ø)`
sei-db/state_db/sc/flatkv/store.go	`59.68% <57.60%> (-4.97%)`	⬇️
... and 1 more

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

cody-littley · 2026-02-24T19:38:27Z

sei-db/db_engine/types.go

+// Checkpointable is an optional capability for DB engines that support
+// efficient point-in-time snapshots via filesystem hardlinks.
+type Checkpointable interface {


Can you augment the godoc with information about concurrency? For example, is it safe to call this method concurrently with updates in other threads? When this method returns, is the checkpoint capable of surviving a host OS crash?

sure, added concurrency and durability documentation to the Checkpointable

cody-littley · 2026-02-25T17:22:53Z

sei-db/state_db/sc/flatkv/snapshot.go

+	snapshotPrefix = "snapshot-"
+	snapshotDirLen = len(snapshotPrefix) + 20
+
+	currentLink    = "current"
+	currentTmpLink = "current-tmp"


Short descriptions for what each constant is for might be helpful.

One way to document this, feel free to push back if you think it's too much overhead.

At past companies, when documenting this sort of file layout, I sometimes find it useful to do the following:

write a simple unit test that generates a basic file structure

pause the unit test before it deletes its data

run tree on the directory created by the test

edit the result for readability copy-paste the rest somewhere

Here's an example: https://github.com/Layr-Labs/eigenda/blob/master/litt/docs/filesystem_layout.md

If the file layout is to big, it might make sense to split it out into a markdown file that you can just reference in the godoc.

good suggestion. adding inline godoc with the directory layout (ASCII tree) above the constants in snapshot.go. this covers the logical structure (which directories exist and what they store).If we add more complexity later (e.g. sharded storage), we may revist and split it out into a dedicated doc.

Yeah, I think that layer document might help people understand FlatKV better, but agree we can work on a ReadMe of this whole FlatKV and include there in the future

cody-littley · 2026-02-25T17:25:29Z

sei-db/state_db/sc/flatkv/snapshot.go

+// not a full path.
+func updateCurrentSymlink(root, snapshotDir string) error {
+	tmpPath := filepath.Join(root, currentTmpLink)
+	_ = os.Remove(tmpPath)


What if this removal fails due to invalid file permissions? I'm guessing you aren't checking the error in case it fails due to the path not existing. Perhaps this can first check if the file exists, delete it if it exists, and then return an error if that deletion fails.

adding error handling here

sei-db/state_db/sc/flatkv/store.go

yzang2019 · 2026-02-26T20:18:27Z

sei-db/config/toml.go

+snapshot-interval = {{ .StateCommit.FlatKVConfig.SnapshotInterval }}
+
+# SnapshotKeepRecent defines how many old snapshots to keep besides the latest one.
+# 0 = keep only the current snapshot. Default: 1.


We probably need more than 1 snapshot here in order to do state export

i see. bump the default to be 2

yzang2019 · 2026-02-26T20:19:25Z

sei-db/config/toml.go

+# SnapshotMinTimeInterval is the minimum wall-clock seconds between consecutive
+# auto-snapshots. Prevents dense snapshots during catch-up. Default: 3600 (1 hour).
+snapshot-min-time-interval = {{ .StateCommit.FlatKVConfig.SnapshotMinTimeInterval }}


I think this is a memiavl specific optimization, and not needed by flatKV, since FlatKV snapshot is so cheap, would rather remove the config for simplicity

agree. I removed snapshot-min-time-interval from FlatKV config/template/code, and now auto-snapshot is triggered only by snapshot-interval.

yzang2019 · 2026-02-26T20:30:01Z

sei-db/state_db/sc/flatkv/snapshot.go

+	snapshotPrefix = "snapshot-"
+	snapshotDirLen = len(snapshotPrefix) + 20
+
+	currentLink    = "current"
+	currentTmpLink = "current-tmp"


Yeah, I think that layer document might help people understand FlatKV better, but agree we can work on a ReadMe of this whole FlatKV and include there in the future

yzang2019 · 2026-02-27T08:43:32Z

sei-db/state_db/sc/flatkv/store_catchup.go

+
+	if replayed > 0 {
+		if !s.config.Fsync {
+			if err := s.flushAllDBs(); err != nil {


Why do we flush here?

during catch-up with NoSync, per-entry batch commits may still be in OS/page cache. If we advance global metadata immediately, a crash can leave the global watermark ahead of durable data.

so we do a single flush before committing global metadata to preserve durability ordering. I added this rationale as an inline comment.

yzang2019 · 2026-02-27T08:44:10Z

sei-db/state_db/sc/flatkv/store_write.go

 }

+// flushAllDBs flushes all data DBs to ensure data is on disk.
+func (s *CommitStore) flushAllDBs() error {


I think the only place where need to call flush is right before we take a new snapshot?

comments above

* main: ERC20 simulation benchmark (#2979) feat(flatkv): add snapshot, WAL catchup, and rollback support (#2972)

## Describe your changes and provide context Introduce a snapshot-based lifecycle for FlatKV so that restarts replay only from the nearest PebbleDB checkpoint instead of the full WAL. Key changes: - Snapshot management: immutable PebbleDB checkpoints created via Checkpoint(), managed through a "current" symlink and atomic directory operations. Configurable interval, retention, and minimum time between snapshots. - Working directory: mutable clone of the baseline snapshot (hardlinks for .sst files) so writes never mutate snapshot dirs. - WAL catchup: on open, replay changelog entries from the snapshot version to the target version using O(1) arithmetic + O(log N) binary search for offset resolution. - Rollback: rewind to the best snapshot <= target, truncate WAL, replay to exact version, and prune future snapshots. - File lock: prevent concurrent access from multiple processes. - Migration: automatically move pre-snapshot flat layout into a versioned snapshot directory on first open. - Auto WAL truncation: periodically discard WAL entries older than the earliest retained snapshot. - Fix account LtHash baseline capture to use pre-batch state when multiple ApplyChangeSets calls precede a single Commit. - Add legacyDB to flushAllDBs. - Mark Iterator/IteratorByPrefix as EXPERIMENTAL (unused in production). ## Testing performed to validate your change

blindchaser added 14 commits February 18, 2026 21:08

feat: flatkv snapshot/catchup

fd40dd9

Merge branch 'main' into yiren/flatkv-snapshot

3d300d8

add legacy

b1dde46

Merge branch 'main' into yiren/flatkv-snapshot

6828d07

refactor

3729907

remove usned func

420e5f4

try lock

7c2b4c3

clean up

43b3051

trigger snapshot

4b1c951

SnapshotMinTimeInterval and SnapshotKeepRecent

d8bd58e

Merge branch 'main' into yiren/flatkv-snapshot

5cf2710

Merge branch 'main' into yiren/flatkv-snapshot

bbe90e5

add iterator comments

bccc292

blindchaser added the non-app-hash-breaking label Feb 24, 2026

blindchaser added 2 commits February 24, 2026 10:17

Merge branch 'main' into yiren/flatkv-snapshot

3f4a371

update comments

dcc76c9

chatgpt-codex-connector bot reviewed Feb 24, 2026

View reviewed changes

fsync

3b0e08f

github-advanced-security bot found potential problems Feb 24, 2026

View reviewed changes

sei-db/state_db/sc/flatkv/snapshot.go

s.pruneSnapshots(dir, version)

success = true

s.lastSnapshotTime = time.Now()

Check warning

Code scanning / CodeQL

Calling the system time Warning

Calling the system time may be a possible source of non-determinism

fix test

1e028d2

blindchaser requested review from cody-littley and yzang2019 February 24, 2026 16:20

blindchaser added 3 commits February 24, 2026 12:15

improve code coverage

04947d1

Merge remote-tracking branch 'origin/main' into yiren/flatkv-snapshot

e4d8596

Merge branch 'main' into yiren/flatkv-snapshot

075dc71

cody-littley approved these changes Feb 25, 2026

View reviewed changes

address comments

c1556f2

Merge branch 'main' into yiren/flatkv-snapshot

7b3b745

yzang2019 reviewed Feb 27, 2026

View reviewed changes

address comments

eeb1a1a

yzang2019 approved these changes Feb 27, 2026

View reviewed changes

Merge remote-tracking branch 'origin/main' into yiren/flatkv-snapshot

8ea2bfe

blindchaser enabled auto-merge (squash) February 27, 2026 18:25

blindchaser merged commit 3990ce9 into main Feb 27, 2026
38 checks passed

blindchaser deleted the yiren/flatkv-snapshot branch February 27, 2026 18:34

yzang2019 added a commit that referenced this pull request Feb 27, 2026

Merge branch 'main' into yzang/refactor-giga-ss

1b711b8

* main: ERC20 simulation benchmark (#2979) feat(flatkv): add snapshot, WAL catchup, and rollback support (#2972)

Conversation

blindchaser commented Feb 24, 2026

Describe your changes and provide context

Testing performed to validate your change

Uh oh!

github-actions bot commented Feb 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector bot Feb 24, 2026

Choose a reason for hiding this comment

Uh oh!

Check warning

codecov bot commented Feb 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

github-actions bot commented Feb 24, 2026 •

edited

Loading

codecov bot commented Feb 24, 2026 •

edited

Loading