Add exporter for FlatKV and CompositeSC by yzang2019 · Pull Request #3064 · sei-protocol/sei-chain

yzang2019 · 2026-03-13T00:07:50Z

Describe your changes and provide context

This PR is adding exporter used by state sync to export live state into snapshot files.

Composite Exporter (composite/exporter.go): Orchestrates the export stream from both the Cosmos (memiavl) and FlatKV backends. All Cosmos modules are exported first, then FlatKV EVM data is appended as a distinct module named "evm_flatkv". The FlatKV phase is only included when WriteMode == SplitWrite or DualWrite.

Composite Importer routing (composite/importer.go): Updated to route the "evm_flatkv" module exclusively to the FlatKV importer, and all other modules exclusively to the Cosmos importer. Previously, EVM data was duplicated to both importers based on the "evm" module name.

FlatKV Importer snapshot fix (flatkv/importer.go): The importer now writes a snapshot after import completes. Import bypasses the WAL and writes directly to PebbleDB in the working directory. Without a snapshot, the next LoadVersion would re-clone the working directory from the pre-import snapshot (e.g. snapshot-0), destroying all imported data.

Testing performed to validate your change

github-actions · 2026-03-13T00:08:51Z

The latest Buf updates on your PR. Results from workflow Buf / buf (pull_request).

Build	Format	Lint	Breaking	Updated (UTC)
`✅ passed`	`✅ passed`	`✅ passed`	`✅ passed`	Mar 17, 2026, 7:46 PM

codecov · 2026-03-13T00:10:38Z

Codecov Report

❌ Patch coverage is 73.14815% with 58 lines in your changes missing coverage. Please review.
✅ Project coverage is 58.50%. Comparing base (4282c07) to head (0176dfb).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
sei-db/state_db/sc/flatkv/exporter.go	74.04%	25 Missing and 9 partials ⚠️
sei-db/state_db/sc/composite/exporter.go	80.95%	5 Missing and 3 partials ⚠️
sei-db/state_db/sc/composite/store.go	41.66%	5 Missing and 2 partials ⚠️
sei-db/state_db/sc/flatkv/store_lifecycle.go	50.00%	3 Missing and 2 partials ⚠️
sei-db/state_db/sc/flatkv/importer.go	0.00%	1 Missing and 1 partial ⚠️
sei-db/state_db/sc/composite/importer.go	87.50%	1 Missing ⚠️
sei-db/tools/cmd/seidb/operations/state_size.go	0.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3064      +/-   ##
==========================================
+ Coverage   58.42%   58.50%   +0.07%     
==========================================
  Files        2088     2090       +2     
  Lines      172108   172238     +130     
==========================================
+ Hits       100552   100763     +211     
+ Misses      62620    62496     -124     
- Partials     8936     8979      +43

Flag	Coverage Δ
sei-chain-pr	`67.82% <73.14%> (?)`
sei-db	`70.41% <ø> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
sei-db/common/evm/keys.go	`91.48% <ø> (ø)`
sei-db/state_db/sc/memiavl/multitree.go	`78.78% <100.00%> (ø)`
sei-db/state_db/ss/composite/store.go	`69.26% <100.00%> (+12.06%)`	⬆️
sei-db/state_db/ss/evm/types.go	`100.00% <ø> (ø)`
sei-db/state_db/sc/composite/importer.go	`92.59% <87.50%> (+92.59%)`	⬆️
sei-db/tools/cmd/seidb/operations/state_size.go	`0.00% <0.00%> (ø)`
sei-db/state_db/sc/flatkv/importer.go	`48.83% <0.00%> (+48.83%)`	⬆️
sei-db/state_db/sc/flatkv/store_lifecycle.go	`61.97% <50.00%> (+5.52%)`	⬆️
sei-db/state_db/sc/composite/store.go	`67.92% <41.66%> (+4.16%)`	⬆️
sei-db/state_db/sc/composite/exporter.go	`80.95% <80.95%> (ø)`
... and 1 more

... and 43 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

* main: feat(flatkv): add comprehensive writing test coverage and centralize account-field semantics (#3057) Fix flaky test caused by async WAL writes (#3063)

sei-db/state_db/sc/composite/store_test.go

blindchaser · 2026-03-13T13:47:15Z

sei-db/state_db/sc/flatkv/exporter.go

+			if err != nil {
+				return nil, fmt.Errorf("open iterator for db %d: %w", e.currentDB, err)
+			}
+			if iter == nil || !iter.First() {


if iter.First() fails because of an iterator/IO error rather than because the DB is empty, we should also handle the iter.Error()?

cody-littley

Would be interested in learning about the import/export lifecycle, and how it's currently working with cosmos. Perhaps a good topic for a white boarding session at the onsite?

cody-littley · 2026-03-13T13:52:26Z

sei-db/state_db/sc/composite/exporter.go

+	}, nil
+}
+
+func (s *SnapshotExporter) Next() (interface{}, error) {


Short godoc explaining the methods in struct would be helpful. Purpose of these methods wasn't immediately obvious based on function names and context.

cody-littley · 2026-03-13T14:13:19Z

sei-db/state_db/sc/flatkv/exporter.go

+	pendingNodes []*types.SnapshotNode
+}
+
+func NewKVExporter(store *CommitStore, version int64) *KVExporter {


Does this sort of export happen on validators on mainnet? If so, would it make sense to have throttles to prevent an export from impacting the handling of regular traffic? Do we currently throttle export from memIAVL?

Export only happens on RPC nodes, validator doesn't export at all

Kbhat1

this would break ss state sync since it isn't aware of the new module evm_flatkv. We can either normalize em_flatkv to evm before sending nodes into ssImporter in rootmulti.restore or update ss/composite.Import to treat both evm and evm_flatkv the same

* main: fix(giga): match v2 correctness checks (#3071) Added clone method to canned random (#3076) Helper files for the flatKV cache implementation (#3072) fix: restore PRs inadvertently reverted by #3039 squash-merge (#3070) Refine logging to avoid printing expensive objects on hot path (#3066) Fix flaky tendermint syncer test (#3065) Add runtime log level control via gRPC admin service (#3062) chore: dcoument run RPC suite on legacy vs giga (#3041) chore: self-contained revert tests, contract reorg, and failure analysis (#3033)

This PR is adding exporter used by state sync to export live state into snapshot files. Composite Exporter (composite/exporter.go): Orchestrates the export stream from both the Cosmos (memiavl) and FlatKV backends. All Cosmos modules are exported first, then FlatKV EVM data is appended as a distinct module named "evm_flatkv". The FlatKV phase is only included when WriteMode == SplitWrite or DualWrite. Composite Importer routing (composite/importer.go): Updated to route the "evm_flatkv" module exclusively to the FlatKV importer, and all other modules exclusively to the Cosmos importer. Previously, EVM data was duplicated to both importers based on the "evm" module name. FlatKV Importer snapshot fix (flatkv/importer.go): The importer now writes a snapshot after import completes. Import bypasses the WAL and writes directly to PebbleDB in the working directory. Without a snapshot, the next LoadVersion would re-clone the working directory from the pre-import snapshot (e.g. snapshot-0), destroying all imported data.

Add exporter for FlatKV and CompositeSC

a16dd03

yzang2019 requested review from blindchaser and cody-littley March 13, 2026 00:08

yzang2019 added the non-app-hash-breaking label Mar 13, 2026

yzang2019 added 4 commits March 12, 2026 17:57

Fix test

5209b6e

Merge branch 'main' into yzang/add-flatkv-export

014a4f4

* main: feat(flatkv): add comprehensive writing test coverage and centralize account-field semantics (#3057) Fix flaky test caused by async WAL writes (#3063)

Add error checking

bbc9d4c

Add more unit test

873f066

blindchaser reviewed Mar 13, 2026

View reviewed changes

sei-db/state_db/sc/composite/store_test.go Outdated Show resolved Hide resolved

blindchaser reviewed Mar 13, 2026

View reviewed changes

blindchaser approved these changes Mar 13, 2026

View reviewed changes

cody-littley approved these changes Mar 13, 2026

View reviewed changes

yzang2019 requested a review from Kbhat1 March 13, 2026 19:57

Kbhat1 requested changes Mar 16, 2026

View reviewed changes

yzang2019 added 4 commits March 16, 2026 15:51

Fix SS import

e71e06d

Add import unit test

340caa5

Address comments

48fbc3b

Kbhat1 approved these changes Mar 17, 2026

View reviewed changes

Merge branch 'main' into yzang/add-flatkv-export

0176dfb

yzang2019 added this pull request to the merge queue Mar 17, 2026

Merged via the queue into main with commit 8ae411c Mar 17, 2026
38 checks passed

yzang2019 deleted the yzang/add-flatkv-export branch March 17, 2026 20:25

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add exporter for FlatKV and CompositeSC#3064

Add exporter for FlatKV and CompositeSC#3064
yzang2019 merged 10 commits intomainfrom
yzang/add-flatkv-export

yzang2019 commented Mar 13, 2026 •

edited

Loading

Uh oh!

github-actions bot commented Mar 13, 2026 •

edited

Loading

Uh oh!

codecov bot commented Mar 13, 2026 •

edited

Loading

Uh oh!

Uh oh!

blindchaser Mar 13, 2026

Uh oh!

cody-littley left a comment •

edited

Loading

Uh oh!

cody-littley Mar 13, 2026

Uh oh!

cody-littley Mar 13, 2026

Uh oh!

yzang2019 Mar 13, 2026

Uh oh!

Kbhat1 left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

yzang2019 commented Mar 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Describe your changes and provide context

Testing performed to validate your change

Uh oh!

github-actions bot commented Mar 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Mar 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

blindchaser Mar 13, 2026

Choose a reason for hiding this comment

Uh oh!

cody-littley left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cody-littley Mar 13, 2026

Choose a reason for hiding this comment

Uh oh!

cody-littley Mar 13, 2026

Choose a reason for hiding this comment

Uh oh!

yzang2019 Mar 13, 2026

Choose a reason for hiding this comment

Uh oh!

Kbhat1 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

yzang2019 commented Mar 13, 2026 •

edited

Loading

github-actions bot commented Mar 13, 2026 •

edited

Loading

codecov bot commented Mar 13, 2026 •

edited

Loading

cody-littley left a comment •

edited

Loading