feat(flatkv): add seidb tooling for flatkv dump state and state analysis#3312
Conversation
|
The latest Buf updates on your PR. Results from workflow Buf / buf (pull_request).
|
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #3312 +/- ##
==========================================
- Coverage 59.18% 59.14% -0.05%
==========================================
Files 2094 2096 +2
Lines 172070 172179 +109
==========================================
- Hits 101847 101834 -13
- Misses 61406 61500 +94
- Partials 8817 8845 +28
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b7280fe19c
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
blindchaser
changed the title
| TotalNumKeys uint64 | ||
| TotalKeySize uint64 | ||
| TotalValueSize uint64 | ||
| TotalSize uint64 |
There was a problem hiding this comment.
Could these be collapsed down into a FlatKVDBSize struct?
| case 0x0a: | ||
| return flatkvBucketAccount | ||
| case 0x07: | ||
| return flatkvBucketCode | ||
| case 0x03: |
There was a problem hiding this comment.
We have constants defined for these, right?
There was a problem hiding this comment.
good point, addressed
3 participants
Summary
This PR adds FlatKV-aware seidb tooling for read-only analysis and dump workflows. The new tooling clones immutable FlatKV snapshots plus changelog state into a temporary store before scanning, so live nodes keep their writer lock while CLI commands operate on a stable view.
sei-db/tools/cmd/seidb/main.go: registersdump-flatkvalongside the existing seidb operations.sei-db/tools/cmd/seidb/operations/flatkv_open.go: selects the latestcurrentsnapshot for height0or the highest snapshot at or below an explicit historical height. The clone path hardlinks snapshot and changelog files, skips liveLOCKfiles, falls back to byte-copy only for cross-device links, retries ENOENT races from concurrent snapshot pruning, and removes the temporary clone on close.sei-db/tools/cmd/seidb/operations/dump_flatkv.go: dumps FlatKV physical keys into ordered bucket files foraccount,code,storage, andlegacy. It preserves physical key bytes, filters optional single-bucket output without changing iterator order, checks iterator errors, and returns final flush/close failures so truncated buffered dumps do not report success.sei-db/tools/cmd/seidb/operations/flatkv_state_size.go: adds FlatKV state-size aggregation with per-DB totals and top EVM contract storage accounting. It classifies physical keys by FlatKV type prefix, emits a dedicatedflatkvDynamoDB row shape, and rejects partial scan results by surfacing iterator errors.sei-db/tools/cmd/seidb/operations/state_size.go: auto-detects sibling FlatKV data or accepts--flatkv-dir, scans FlatKV only for compatible module filters, prints FlatKV totals next to memIAVL totals, and skips FlatKV export when open or scan errors would make the result incomplete.Test plan
sei-db/tools/cmd/seidb/operations/dump_flatkv_test.go: covers all-bucket dumps, single-bucket filtering, dump file headers, physical key/value formatting, and bucket validation.sei-db/tools/cmd/seidb/operations/flatkv_state_size_test.go: covers physical key classification, contract address extraction, per-DB size accounting, top-contract ordering, and DynamoDB analysis row shape.sei-db/tools/cmd/seidb/operations/flatkv_open_test.go: covers latest snapshot selection fromcurrent, explicit historical snapshot selection, missing current and missing snapshot failures, changelog cloning, lock-file exclusion, retry-on-ENOENT behavior, and real read-only opens for latest and historical heights.gofmt -s -landgo test ./sei-db/tools/cmd/seidb/operations.