Iterator utilities and simplfications by cody-littley · Pull Request #3513 · sei-protocol/sei-chain

cody-littley · 2026-05-27T21:03:16Z

Describe your changes and provide context

This PR does prep work for flatKV iteration and iteration over modules during migration.

Interface simplfiication

We had a large number of iterator interfaces defined in different packages. Each iterator interface did almost the same thing, but was slightly different. Since we need to construct iterators that walk over the data in multiple packages, this was making things overlay complex.

I simplified things by selecting the the iterator interface used by composite.Store and making that interface the one that all DBs implement.

Iterator utilities

In order handle iteration on flatKV and on stores in the middle of migration, we need ways to combine and modify iterators. Now that all of the iterators have the same type, we can write these utilities generically and reuse them, as opposed to building a dozen structs that do almost the same thing. There are two new utility structs introduced by this PR:

merging iterator: takes N child iterators and combines them into a single unified iterator
mapping iterator: takes a child iterator and maps (key, value) pairs into different (key, value) pairs

In this PR, I used those iterators to re-implement flatKV's exporter. In the next PR, I will also use these utilities to implement cross-DB and mid-migration iteration.

Testing performed to validate your change

unit tests

cursor · 2026-05-27T21:04:06Z

PR Summary

Medium Risk
Wide refactor of iterator contracts across state/MVCC/FlatKV paths changes iteration semantics (especially FlatKV global order and Pebble auto-positioning); mistakes could affect exports, dumps, and future migration iteration despite solid unit tests.

Overview
This PR unifies database iteration on Tendermint’s dbm.Iterator by removing the separate KeyValueDBIterator, DBIterator, and FlatKV Iterator types from db_engine/types and related packages. KeyValueDB.NewIter, StateStore forward/reverse iterators, MVCC/Pebble/RocksDB wrappers, mocks, and tools now all return dbm.Iterator.

It adds reusable mapping and merging iterators under sei-db/common/iterators/ (filter/remap entries; merge sorted children in lex order with last child winning on duplicate keys), with unit tests covering skips, errors, shared key buffers, and close semantics.

FlatKV RawGlobalIterator is reimplemented on those utilities: per–data-DB iterators skip _meta/* keys, then merge into global lexicographic order (replacing the deleted sequential store_iterator). The API becomes (dbm.Iterator, error); exporter, dump/size CLIs, migration docs, and tests follow the new contract and the for ; iter.Valid(); iter.Next() loop style.

The thin Pebble KeyValueDB iterator now implements only dbm.Iterator (including Domain and positioning at First() on open). MVCC iterators still implement dbm.Iterator while keeping their version-resolution Next() behavior.

^{Reviewed by Cursor Bugbot for commit 9948bbb. Bugbot is set up for automated code reviews on this repo. Configure here.}

github-actions · 2026-05-27T21:04:54Z

The latest Buf updates on your PR. Results from workflow Buf / buf (pull_request).

Build	Format	Lint	Breaking	Updated (UTC)
`✅ passed`	`✅ passed`	`✅ passed`	`✅ passed`	May 28, 2026, 4:00 PM

codecov · 2026-05-27T21:07:28Z

Codecov Report

❌ Patch coverage is 74.39446% with 74 lines in your changes missing coverage. Please review.
✅ Project coverage is 58.66%. Comparing base (1b322f0) to head (9948bbb).

Files with missing lines	Patch %	Lines
sei-db/common/iterators/merging_iterator.go	77.96%	16 Missing and 10 partials ⚠️
sei-db/common/iterators/mapping_iterator.go	77.50%	10 Missing and 8 partials ⚠️
sei-db/state_db/sc/flatkv/store_read.go	40.74%	12 Missing and 4 partials ⚠️
sei-db/tools/cmd/seidb/operations/dump_flatkv.go	63.63%	2 Missing and 2 partials ⚠️
sei-db/state_db/sc/flatkv/exporter.go	50.00%	2 Missing and 1 partial ⚠️
sei-db/db_engine/pebbledb/iterator.go	91.30%	1 Missing and 1 partial ⚠️
sei-db/db_engine/pebbledb/mvcc/db_ascending.go	0.00%	2 Missing ⚠️
...db/tools/cmd/seidb/operations/flatkv_state_size.go	60.00%	1 Missing and 1 partial ⚠️
sei-db/state_db/ss/evm/store.go	50.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3513      +/-   ##
==========================================
- Coverage   59.04%   58.66%   -0.38%     
==========================================
  Files        2199     2164      -35     
  Lines      182096   178455    -3641     
==========================================
- Hits       107510   104690    -2820     
+ Misses      64935    64351     -584     
+ Partials     9651     9414     -237

Flag	Coverage Δ
sei-chain-pr	`66.63% <78.13%> (?)`
sei-db	`70.41% <100.00%> (ø)`
sei-db-state-db	`?`
sei-db-state-db-pr	`75.76% <50.00%> (?)`

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
sei-db/db_engine/dbcache/cached_key_value_db.go	`76.08% <100.00%> (ø)`
sei-db/db_engine/pebbledb/db.go	`72.07% <100.00%> (ø)`
sei-db/db_engine/pebbledb/mvcc/db.go	`65.56% <100.00%> (ø)`
sei-db/db_engine/pebbledb/mvcc/iterator.go	`66.81% <ø> (ø)`
...i-db/db_engine/pebbledb/mvcc/iterator_ascending.go	`0.00% <ø> (ø)`
sei-db/db_engine/rocksdb/mvcc/db.go	`59.83% <100.00%> (ø)`
sei-db/db_engine/rocksdb/mvcc/iterator.go	`86.86% <ø> (ø)`
sei-db/db_engine/types/types.go	`100.00% <ø> (ø)`
sei-db/ledger_db/receipt/tx_hash_index.go	`64.66% <100.00%> (ø)`
sei-db/state_db/sc/flatkv/verify.go	`48.71% <100.00%> (ø)`
... and 11 more

... and 37 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

blindchaser · 2026-05-27T21:54:01Z

+	}, keys)
+}
+
+func TestMergingIterator_DuplicateKeys(t *testing.T) {


can we have a comment in the func NewMergingIterator describe the intentional duplicate keys behavior

You make a very good point.

Although not broken for the flatKV raw iterator, the current merge behavior in this branch was actually incorrect. It currently emits duplicate keys if they show up in multiple child iterators, but it really should be de-duping. This will be necessary for future changes that come after this PR.

I've fixed this bug and documented the behavior.

blindchaser · 2026-05-27T21:56:51Z

+
+// NewMappingIterator returns an iterator that emits remapped key/value pairs
+// from parent, skipping pairs for which remapper returns skip=true.
+func NewMappingIterator(parent dbm.Iterator, remapper IteratorRemapper) *mappingIterator {


do we want to return *mappingIterator or dbm.Iterator as other constructors

Changed return type to dbm.Iterator

blindchaser · 2026-05-27T22:24:24Z

 }
 func (f *failingEVMStore) Has(string, []byte) bool                { return false }
-func (f *failingEVMStore) RawGlobalIterator() flatkv.Iterator     { return nil }
+func (f *failingEVMStore) RawGlobalIterator() dbm.Iterator        { return nil }


right now RawGlobalIterator() dbm.Iterator swallows construction errors into an invalidIterator whose Error() reports the cause. This works only if every caller remembers to check Error() after the loop, a missed check makes a failed iterator open indistinguishable from an empty store.

how about we change the signature to RawGlobalIterator() (dbm.Iterator, error). would also let us drop iterators.NewInvalidIterator since looks it has no other production user

Good suggestion, done.

I really dislike the dbm.Iterator API. It would be MUCH better to return errors when they actually happen, instead of hoping that people remember to call the Error() method. I want to keep the scope of this PR small. But in the future, my plan is to change the API to something more sensible. That should be a lot easier in the future, since we're refactoring things so that everything uses the same iterator interface.

cursor

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

There are 2 total unresolved issues (including 1 from previous review).

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit d673ffd. Configure here.}

Cody Littley added 3 commits May 27, 2026 15:03

simplify iterator interfaces

5baaa02

implemented merging iterator

e96bd93

simplify types, add iterator utilities

99ef64c

cody-littley requested review from Kbhat1 and blindchaser May 27, 2026 21:03

cody-littley self-assigned this May 27, 2026

cody-littley added the non-app-hash-breaking label May 27, 2026

cursor Bot reviewed May 27, 2026

View reviewed changes

Comment thread sei-db/common/iterators/mapping_iterator.go

blindchaser reviewed May 27, 2026

View reviewed changes

Kbhat1 approved these changes May 28, 2026

View reviewed changes

made suggested changes

caf43db

blindchaser approved these changes May 28, 2026

View reviewed changes

cursor Bot reviewed May 28, 2026

View reviewed changes

Comment thread sei-db/common/iterators/merging_iterator.go

Comment thread sei-db/common/iterators/invalid_iterator.go Outdated

Cody Littley added 2 commits May 28, 2026 08:57

made suggested changes

f01b9b0

fix compile issue

155cc8b

cursor Bot reviewed May 28, 2026

View reviewed changes

Comment thread sei-db/common/iterators/merging_iterator.go

fix lint

d85e888

cursor Bot reviewed May 28, 2026

View reviewed changes

Comment thread sei-db/common/iterators/mapping_iterator.go

Comment thread sei-db/common/iterators/merging_iterator.go

made suggested changes

d673ffd

cursor Bot reviewed May 28, 2026

View reviewed changes

Comment thread sei-db/common/iterators/mapping_iterator_test.go

Cody Littley added 3 commits May 28, 2026 10:35

fix broken test

5460987

Merge branch 'main' into cjl/simple-iterator-type

daf6c94

made suggested change

9948bbb

cody-littley added this pull request to the merge queue May 28, 2026

Merged via the queue into main with commit 8567422 May 28, 2026
54 checks passed

cody-littley deleted the cjl/simple-iterator-type branch May 28, 2026 18:10

Uh oh!

Conversation

cody-littley commented May 27, 2026

Describe your changes and provide context

Interface simplfiication

Iterator utilities

Testing performed to validate your change

Uh oh!

cursor Bot commented May 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR Summary

Uh oh!

github-actions Bot commented May 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov Bot commented May 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

blindchaser May 27, 2026

Choose a reason for hiding this comment

Uh oh!

cody-littley May 28, 2026

Choose a reason for hiding this comment

Uh oh!

blindchaser May 27, 2026

Choose a reason for hiding this comment

Uh oh!

cody-littley May 28, 2026

Choose a reason for hiding this comment

Uh oh!

blindchaser May 27, 2026

Choose a reason for hiding this comment

Uh oh!

cody-littley May 28, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

cursor Bot commented May 27, 2026 •

edited

Loading

github-actions Bot commented May 27, 2026 •

edited

Loading

codecov Bot commented May 27, 2026 •

edited

Loading