Restructure sei data folder for Giga by yzang2019 · Pull Request #3155 · sei-protocol/sei-chain

yzang2019 · 2026-03-31T23:44:33Z

Describe your changes and provide context

The current data folder is not intuitive and not very clean for Giga storage. It currently looks like this:

The goal of this PR is to restructure the data folder for Giga into this new structure while also making it backward compatible:

Testing performed to validate your change

Added unit test to make sure it works for both new nodes and existing nodes which has the old path

github-actions · 2026-03-31T23:45:29Z

The latest Buf updates on your PR. Results from workflow Buf / buf (pull_request).

Build	Format	Lint	Breaking	Updated (UTC)
`✅ passed`	`✅ passed`	`✅ passed`	`✅ passed`	Apr 22, 2026, 5:01 AM

codecov · 2026-03-31T23:58:05Z

Codecov Report

❌ Patch coverage is 92.03540% with 9 lines in your changes missing coverage. Please review.
✅ Project coverage is 58.77%. Comparing base (f3d84e0) to head (8ebf801).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
sei-tendermint/cmd/tendermint/commands/reset.go	76.66%	5 Missing and 2 partials ⚠️
...endermint/cmd/tendermint/commands/reindex_event.go	75.00%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3155      +/-   ##
==========================================
+ Coverage   58.50%   58.77%   +0.26%     
==========================================
  Files        2072     2101      +29     
  Lines      207965   175825   -32140     
==========================================
- Hits       121680   103334   -18346     
+ Misses      77494    63325   -14169     
- Partials     8791     9166     +375

Flag	Coverage Δ
sei-chain-pr	`61.65% <91.66%> (?)`
sei-db	`69.36% <ø> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
app/receipt_store_config.go	`70.00% <100.00%> (ø)`
sei-db/common/utils/path.go	`100.00% <100.00%> (ø)`
sei-db/config/ss_config.go	`100.00% <ø> (ø)`
sei-db/state_db/sc/composite/store.go	`64.83% <100.00%> (ø)`
sei-db/state_db/sc/memiavl/store.go	`90.90% <100.00%> (ø)`
sei-db/state_db/ss/composite/store.go	`79.39% <100.00%> (+11.10%)`	⬆️
sei-tendermint/config/config.go	`72.60% <100.00%> (+3.88%)`	⬆️
sei-tendermint/config/db.go	`100.00% <100.00%> (ø)`
...endermint/cmd/tendermint/commands/reindex_event.go	`73.65% <75.00%> (+0.25%)`	⬆️
sei-tendermint/cmd/tendermint/commands/reset.go	`58.44% <76.66%> (+14.61%)`	⬆️

... and 1801 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

blindchaser · 2026-04-01T14:57:38Z

+	return filepath.Join(baseDir, subDir)
+}
+
+func pathExists(path string) bool {


dup func pathExists

You mean this one and the one in sei-db? I think this is fine we don't wanna couple sei-tendemrint and sei-db. Tendermint manages their own db separately than sei-db

blindchaser · 2026-04-01T14:59:17Z

-	if !os.FileExists(filepath.Join(cfg.DBDir(), "blockstore.db")) {
+	blockstoreDir := tmcfg.ResolveDBDir("blockstore", cfg.DBDir())
+	if !os.FileExists(filepath.Join(blockstoreDir, "blockstore.db")) {
 		return nil, nil, fmt.Errorf("no blockstore found in %v", cfg.DBDir())


the error msg seems to should be against blockstoreDir instead of cfg.DBDir()

good catch!

blindchaser · 2026-04-01T14:59:56Z

-		return nil, nil, fmt.Errorf("no blockstore found in %v", cfg.DBDir())
+	stateDir := tmcfg.ResolveDBDir("state", cfg.DBDir())
+	if !os.FileExists(filepath.Join(stateDir, "state.db")) {
+		return nil, nil, fmt.Errorf("no state store found in %v", cfg.DBDir())


same as above, error msg against stateDir instead of cfg.DBDir()

good catch!

blindchaser · 2026-04-01T15:12:04Z

-	}
+// ResetState removes all blocks, tendermint state, indexed transactions and evidence.
+// It handles both the legacy flat layout and the new subdirectory layout.
+func ResetState(dbDir string) error {


consider updating reset related unit testings

blindchaser · 2026-04-01T15:15:50Z

 	}
 	if receiptConfig.DBDirectory == "" {
-		receiptConfig.DBDirectory = filepath.Join(homePath, "data", "receipt.db")
+		receiptConfig.DBDirectory = utils.GetReceiptStorePath(homePath)


readReceiptStoreConfig() was changed to default to data/ledger/receipt.db, but the emitted config template still says /data/receipt.db

need to update the related toml.go

* main: plt-228 fixed static check on app and evmrpc package (#3154) flatkv cache (#3027) Make cryptosim state store backend configurable + No Op Wrapper + Read Disable Config (#3145) Add warning message for IAVL deprecation (#3159) Change default min valid per window to zero (#3157) support for starting autobahn from non-zero global block (#3136) Fix upgrade list comparison to respect semver (#3153)

masih

No blockers 🚀

masih · 2026-04-02T09:10:20Z

 func GetStateStorePath(homePath string, backend string) string {
-	return filepath.Join(homePath, "data", backend)
+	legacyPath := filepath.Join(homePath, "data", backend)
+	if PathExists(legacyPath) {


Minor: Is there ever a case where an abrupt node failure could cause the creation of an empty directory here?

If so, for robustness i recommend checking if this dir is empty and proceed with legacy path iff it is not empty.

If the creation of empty dir happens, then it will continue proceed with the old path, which is fine. I don't think there's any case where it is started with the new path first and then suddenly crash and create an old path

masih · 2026-04-02T09:12:49Z

-	return filepath.Join(homePath, "data", "committer.db")
+// PathExists returns true if the given path exists on disk.
+func PathExists(path string) bool {
+	_, err := os.Stat(path)


Harden this by requiring the type of path? e.g. directory? file etc.

This is racy in that it doesn't atomically check and it repeatedly checks but hopefully the probability of race is low enough that we can accept the race condition as negligible.

Make sense, will do

cody-littley

Does this PR handle migration (i.e. moving memiavl data from original location to new location)?

yzang2019 · 2026-04-03T16:41:15Z

Does this PR handle migration (i.e. moving memiavl data from original location to new location)?

We can handle that in the future, this PR doesn't do the migration, it's preparing the new folder structure for new nodes only

* main: (51 commits) Giga storage integration test (#3268) test(flatkv): add flatkv integration testings (#3262) perf(app): reuse decoded transactions across ProcessProposalHandler hot path (#3257) Fix of the proto conv testing (#3261) FlatKV refactor for state sync import + export (#3250) Validate block part index matches proof index (CON-20) (#3256) fix: add retry to apt-get update in Docker CI (#3264) fix: autobahn InitChain, GetValidators, and mempool TTL (CON-249) (#3243) Fix buffer offset in ProposerPriorityHash (CON-200) (#3255) Handle error case in light client divergence detector (#3254) perf(evmrpc): eliminate redundant block fetches in simulate backend (#3208) fix(evmrpc): omit notifications from legacy JSON-RPC batch responses per spec (#3246) fix: deduplicate block fetch in getTransactionReceipt (#3244) Made autobahn producer use TxMempool (#3224) Skip signature event building during Cosmos CheckTx/ReCheckTx (#3230) Regenerate changelog in prep to tag v6.4.2 (#3240) Fix receipt default retention (#3237) feat(flatkv): introduce module-prefix physical keys across all FlatKV DBs (#3229) added a ProposerAddress check to setProposal CON-250 (#3232) feat: add AUTOBAHN option to local docker cluster (CON-247) (#3220) ...

* main: Giga store migration guide (#3227) Update SS write/read modes into single evm-ss-mode (#3278) fix(evmrpc): return empty array instead of null for eth_getFilterLogs and eth_getFilterChanges (#3292) feat: add autobahn integration tests - batch 1 (CON-249) (#3234) add block height modified to legacy data (#3276) Export ProposerPriorityHash metric for divergence monitoring (#3277) backport CW/wasmd@76eaff4 (#3258)

Restructure sei data folder for Giga

7c6ac8f

yzang2019 requested review from Kbhat1, blindchaser, cody-littley and masih March 31, 2026 23:45

yzang2019 added the non-app-hash-breaking label Mar 31, 2026

yzang2019 requested a review from jewei1997 March 31, 2026 23:45

Fix unit test

88daf32

blindchaser reviewed Apr 1, 2026

View reviewed changes

Change method name

0df74b4

blindchaser reviewed Apr 1, 2026

View reviewed changes

blindchaser approved these changes Apr 1, 2026

View reviewed changes

Kbhat1 approved these changes Apr 1, 2026

View reviewed changes

yzang2019 and others added 3 commits April 1, 2026 13:07

Fix path

23d7a91

Merge branch 'main' into yzang/restruct-data-folder

b4d61bb

masih approved these changes Apr 2, 2026

View reviewed changes

cody-littley approved these changes Apr 2, 2026

View reviewed changes

jewei1997 approved these changes Apr 2, 2026

View reviewed changes

Merge branch 'main' into yzang/restruct-data-folder

022fd22

yzang2019 and others added 2 commits April 3, 2026 09:51

Change to dirExist

3ebd0ae

Merge branch 'main' into yzang/restruct-data-folder

343d67d

yzang2019 enabled auto-merge April 3, 2026 17:01

yzang2019 added this pull request to the merge queue Apr 3, 2026

github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Apr 3, 2026

yzang2019 enabled auto-merge April 20, 2026 20:33

yzang2019 added 2 commits April 21, 2026 21:37

Fix unit test

8ebf801

yzang2019 added this pull request to the merge queue Apr 22, 2026

Merged via the queue into main with commit 8a0da0e Apr 22, 2026
40 of 41 checks passed

yzang2019 deleted the yzang/restruct-data-folder branch April 22, 2026 09:21

Uh oh!

Conversation

yzang2019 commented Mar 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Describe your changes and provide context

Testing performed to validate your change

Uh oh!

github-actions Bot commented Mar 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov Bot commented Mar 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

masih left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cody-littley left a comment

Choose a reason for hiding this comment

Uh oh!

yzang2019 commented Apr 3, 2026

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

yzang2019 commented Mar 31, 2026 •

edited

Loading

github-actions Bot commented Mar 31, 2026 •

edited

Loading

codecov Bot commented Mar 31, 2026 •

edited

Loading