fix: pruning goroutine lifecycle and prune failure snapshot by blindchaser · Pull Request #2800 · sei-protocol/sei-chain

blindchaser · 2026-02-04T19:04:30Z

Describe your changes and provide context

Wrap StateStore with PrunableStateStore that stops the pruning goroutine before closing the underlying database
Remove redundant stateStore.Close() from HandleClose() since cms.Close() already handles it
Add nil checks in Prune() methods for pebbledb/rocksdb as an additional safety layer
Use sync.Once to ensure Start/Stop/Close operations are safe to call multiple times
Prune old memiavl snapshots even when snapshot rewrite fails

Testing performed to validate your change

github-actions · 2026-02-04T19:05:30Z

The latest Buf updates on your PR. Results from workflow Buf / buf (pull_request).

Build	Format	Lint	Breaking	Updated (UTC)
`✅ passed`	`✅ passed`	`✅ passed`	`✅ passed`	Feb 21, 2026, 10:44 PM

codecov · 2026-02-04T19:13:13Z

Codecov Report

❌ Patch coverage is 71.42857% with 16 lines in your changes missing coverage. Please review.
✅ Project coverage is 57.28%. Comparing base (1bcbb0e) to head (f5da788).
⚠️ Report is 20 commits behind head on main.

Files with missing lines	Patch %	Lines
sei-db/state_db/ss/composite/store.go	60.00%	4 Missing and 2 partials ⚠️
sei-db/state_db/ss/pruning/manager.go	88.57%	2 Missing and 2 partials ⚠️
sei-db/db_engine/pebbledb/mvcc/db.go	0.00%	1 Missing and 1 partial ⚠️
sei-db/db_engine/rocksdb/mvcc/db.go	0.00%	1 Missing and 1 partial ⚠️
sei-db/state_db/sc/memiavl/db.go	0.00%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2800      +/-   ##
==========================================
- Coverage   57.30%   57.28%   -0.03%     
==========================================
  Files        2089     2089              
  Lines      172183   172210      +27     
==========================================
- Hits        98669    98649      -20     
- Misses      64709    64742      +33     
- Partials     8805     8819      +14

Flag	Coverage Δ
sei-chain	`52.77% <74.07%> (-0.02%)`	⬇️
sei-cosmos	`48.15% <ø> (ø)`
sei-db	`68.42% <0.00%> (-0.31%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
app/app.go	`71.24% <ø> (+0.14%)`	⬆️
sei-db/db_engine/pebbledb/mvcc/db.go	`63.92% <0.00%> (-0.23%)`	⬇️
sei-db/db_engine/rocksdb/mvcc/db.go	`57.31% <0.00%> (-0.47%)`	⬇️
sei-db/state_db/sc/memiavl/db.go	`65.63% <0.00%> (-0.21%)`	⬇️
sei-db/state_db/ss/pruning/manager.go	`90.69% <88.57%> (+8.87%)`	⬆️
sei-db/state_db/ss/composite/store.go	`49.69% <60.00%> (+0.62%)`	⬆️

... and 31 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

sei-db/state_db/ss/pruning/manager.go

+		default:
+		}
+
+		pruneStartTime := time.Now()


sei-db/state_db/ss/pruning/manager.go

-				}
+	m.startOnce.Do(func() {
+		m.wg.Add(1)
+		go m.pruneLoop()


sei-db/state_db/ss/pruning/manager.go

-			time.Sleep(time.Duration(m.pruneInterval+randomDelay) * time.Second)
+		// Generate a random percentage (between 0% and 100%) of the fixed interval as a delay
+		randomPercentage := rand.Float64()
+		randomDelay := int64(float64(m.pruneInterval) * randomPercentage)


… Close - Add proper Stop() with stopCh/WaitGroup to pruning Manager for graceful shutdown - Save pruning manager in CompositeStateStore and stop it on Close (idempotent) - Add defensive nil checks in pebbledb/rocksdb Prune() for closed databases - Remove duplicate stateStore.Close() in app HandleClose (already closed by BaseApp) Co-authored-by: Cursor <cursoragent@cursor.com>

Previously, pruneSnapshots() was only called after a successful snapshot rewrite. If snapshot creation kept failing, old snapshots would accumulate indefinitely. Now we prune old snapshots regardless of rewrite outcome. Co-authored-by: Cursor <cursoragent@cursor.com>

cody-littley

LGTM.

What's the reason for the random delay in the pruning loop? Is the idea to ensure that multiple DBs don't always wake up and prune at the same time?

blindchaser · 2026-02-18T17:48:10Z

LGTM.

What's the reason for the random delay in the pruning loop? Is the idea to ensure that multiple DBs don't always wake up and prune at the same time?

exactly, without this they'd all wake up and hit disk at the same time, may causing I/O spikes

blindchaser · 2026-02-21T22:46:06Z

/backport

seidroid · 2026-02-21T22:46:48Z

Created backport PR for release/v6.3:

Backport release/v6.3: fix: pruning goroutine lifecycle and prune failure snapshot #2947 with remaining conflicts!

Please cherry-pick the changes locally and resolve any conflicts.

git fetch origin backport-2800-to-release/v6.3
git worktree add --checkout .worktree/backport-2800-to-release/v6.3 backport-2800-to-release/v6.3
cd .worktree/backport-2800-to-release/v6.3
git reset --hard HEAD^
git cherry-pick -x c6621a327deee051f79f44fbce328c624ea80ccf
git push --force-with-lease

- Wrap StateStore with PrunableStateStore that stops the pruning goroutine before closing the underlying database - Remove redundant stateStore.Close() from HandleClose() since cms.Close() already handles it - Add nil checks in Prune() methods for pebbledb/rocksdb as an additional safety layer - Use sync.Once to ensure Start/Stop/Close operations are safe to call multiple times - Prune old memiavl snapshots even when snapshot rewrite fails --------- Co-authored-by: Cursor <cursoragent@cursor.com> (cherry picked from commit c6621a3)

…ailure snapshot (#2947) Backport of #2800 to `release/v6.3`. --------- Co-authored-by: yirenz <blindchaser@users.noreply.github.com> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: blindchaser <zengyiren0@gmail.com>

- Wrap StateStore with PrunableStateStore that stops the pruning goroutine before closing the underlying database - Remove redundant stateStore.Close() from HandleClose() since cms.Close() already handles it - Add nil checks in Prune() methods for pebbledb/rocksdb as an additional safety layer - Use sync.Once to ensure Start/Stop/Close operations are safe to call multiple times - Prune old memiavl snapshots even when snapshot rewrite fails --------- Co-authored-by: Cursor <cursoragent@cursor.com>

blindchaser added the non-app-hash-breaking label Feb 4, 2026

github-advanced-security bot found potential problems Feb 4, 2026

View reviewed changes

blindchaser force-pushed the yiren/fix-prune branch 3 times, most recently from b1ef012 to 9dba675 Compare February 17, 2026 15:49

blindchaser force-pushed the yiren/fix-prune branch from 9dba675 to c4fa85d Compare February 17, 2026 15:51

blindchaser and others added 2 commits February 17, 2026 11:06

Merge branch 'main' into yiren/fix-prune

231d6ad

blindchaser changed the title ~~fix: pruning goroutine lifecycle management to prevent nil pointer on Close~~ fix: pruning goroutine lifecycle and prune failure snapshot Feb 18, 2026

Merge branch 'main' into yiren/fix-prune

051541e

blindchaser requested review from cody-littley, jewei1997 and yzang2019 February 18, 2026 14:48

cody-littley approved these changes Feb 18, 2026

View reviewed changes

Merge branch 'main' into yiren/fix-prune

0296402

yzang2019 approved these changes Feb 18, 2026

View reviewed changes

blindchaser added 2 commits February 18, 2026 14:50

Merge branch 'main' into yiren/fix-prune

ac10ad9

Merge branch 'main' into yiren/fix-prune

f5da788

blindchaser merged commit c6621a3 into main Feb 18, 2026
55 of 58 checks passed

blindchaser deleted the yiren/fix-prune branch February 18, 2026 20:54

blindchaser added the backport release/v6.3 label Feb 21, 2026

seidroid bot mentioned this pull request Feb 21, 2026

Backport release/v6.3: fix: pruning goroutine lifecycle and prune failure snapshot #2947

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: pruning goroutine lifecycle and prune failure snapshot#2800

fix: pruning goroutine lifecycle and prune failure snapshot#2800
blindchaser merged 7 commits intomainfrom
yiren/fix-prune

blindchaser commented Feb 4, 2026 •

edited

Loading

Uh oh!

github-actions bot commented Feb 4, 2026 •

edited

Loading

Uh oh!

codecov bot commented Feb 4, 2026 •

edited

Loading

Uh oh!

Check warning

Check notice

Check notice

cody-littley left a comment

Uh oh!

blindchaser commented Feb 18, 2026

Uh oh!

Uh oh!

blindchaser commented Feb 21, 2026

Uh oh!

seidroid bot commented Feb 21, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

blindchaser commented Feb 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Describe your changes and provide context

Testing performed to validate your change

Uh oh!

github-actions bot commented Feb 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Feb 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Check warning

Check notice

Check notice

cody-littley left a comment

Choose a reason for hiding this comment

Uh oh!

blindchaser commented Feb 18, 2026

Uh oh!

Uh oh!

blindchaser commented Feb 21, 2026

Uh oh!

seidroid bot commented Feb 21, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

blindchaser commented Feb 4, 2026 •

edited

Loading

github-actions bot commented Feb 4, 2026 •

edited

Loading

codecov bot commented Feb 4, 2026 •

edited

Loading