perf: consolidate WorldDiff (address,key) maps to cut verifier memory#105
Merged
Conversation
Follow-up to #104 (shamatar's suggestion). Merge `read_storage_slots`, `written_storage_slots`, and `committed_reads_at_depth_zero` — three `RollbackableSet<(H160, U256)>` — into one `RollbackableMap<(H160, U256), u8>` of access flags, storing the 52-byte `(address, key)` once instead of 3x. External-rollback semantics and the public `committed_reads_at_depth_zero_iter` accessor are preserved. 54 lib tests pass (incl. the storage-log trace tests). Measured on the eravm-airbender-verifier guest: the worst-case 67xxx production batch drops from needing ~920-952 MiB to fitting in 768 MiB (~150+ MiB saved), on top of #104. This is "Group B" (membership sets). "Group A" (storage_changes / paid_changes / transient_storage_changes) is a separate, hot-path change to evaluate next. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
0xVolosnikov
force-pushed
the
vv/merge-slot-maps
branch
from
f999edf to
b88cdde
Compare
>⚠️ **Experiment — not for merge.** Posted to share a measured result. Stacked on #105 (`vv/merge-slot-maps`); the diff here is just the Group A commit. ## What "Group A" of #104's map-consolidation idea: merge the two **internal-rollback** `(address, key)` maps — `storage_changes` (`U256`) and `paid_changes` (`u32`) — into one `RollbackableMap<(H160, U256), StorageWriteEntry { value, paid }>`, storing the 52-byte key once instead of twice. `transient_storage_changes` is deliberately **left separate** (distinct keyspace — merging it would only add an always-absent field to every persistent entry). ## Correctness - 54 lib tests pass (incl. the `storage_changes_*` proptests and the Boojum trace tests). - Validated end-to-end on the matter-labs/eravm-airbender-verifier guest: batch 67912 produces the **identical commitment / proof public input** as baseline. - One subtlety handled: `get_storage_changes_after` must *not* filter `before == after` — a paid update only happens within a `write_storage` call alongside the value write, and `changes_after` coalesces by key, so there is no standalone "paid-only" change. (A proptest caught an earlier wrong filter.) ## Measured saving On the eravm guest, worst-case 67912 fits at **720 MiB** vs **768 MiB** with #105 (Group B) alone → **~48 MiB additional** (cumulative B+A ≈ ~200 MiB off guest peak). ## Cost / why it's an experiment, not a ready PR - `get_storage_state()` changes return type to `&BTreeMap<_, StorageWriteEntry>`, so **every consumer must project `.value`** (1 site here in `tracing.rs`; 2 in the eravm verifier's `vm_fast`). API ripple. - It sits on the **SLOAD/SSTORE hot path**, so it needs **shadow-mode validation** against the legacy VM before it could land. - ROI is lower than #105: ~⅓ the saving for more surface area and risk. **Recommendation:** hold unless a concrete memory need remains after #105 + the guest heap bump. Opening as a draft so the data and approach are on record. Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
- re-export `StorageWriteEntry` from the crate root (it's now part of the public `get_storage_state()` return type) [P2] - write_storage: one insert per path instead of two for non-free writes — compute the final paid amount, then insert once [P3] - add a non-zero-cost test (`merged_storage_write_tracks_paid_and_rolls_back`) covering prior_paid/prepaid + rollback of the merged entry [P3] - refresh stale doc comments referencing the old `storage_changes` / `committed_reads_at_depth_zero` field names; document StorageWriteEntry fields Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
0xVolosnikov
changed the title
0xVolosnikov
added a commit
that referenced
this pull request
Jul 2, 2026
…#105) Reduces `WorldDiff` memory on large mainnet batches (consumer: matter-labs/eravm-airbender-verifier#18) by removing duplicated `(address, key)` keys across its maps. **Stacked on #104** (`vv/memopt-on-popzxc`). > **Scope note:** this PR now contains **both** consolidation steps — the experimental "Group A" (#106) was merged into this branch, so it's no longer separate. Both are described below. ## Changes - **Group B — merge the three membership sets** (`read_storage_slots`, `written_storage_slots`, `committed_reads_at_depth_zero`) into one `slot_flags: RollbackableMap<(H160,U256), u8>` of bit flags. External-rollback semantics preserved; public `committed_reads_at_depth_zero_iter` kept (now filters the flag). - **Group A — merge the two internal-rollback write maps** (`storage_changes: U256` + `paid_changes: u32`) into one `storage_writes: RollbackableMap<(H160,U256), StorageWriteEntry { value, paid }>`. `transient_storage_changes` left separate (distinct keyspace). ##⚠️ Public API change `WorldDiff::get_storage_state()` now returns `&BTreeMap<_, StorageWriteEntry>` (was `…, U256>`), and `StorageWriteEntry` is re-exported from the crate root. **Direct callers must project `.value`.** The downstream consumer is eravm-airbender-verifier's `vm_fast` (2 sites) — its vm2 pin bump must land together with that `.value` projection. ## Correctness - Rollback groups unchanged (`slot_flags` external; `storage_writes` internal). - `write_storage` does a single insert per path; `prepaid` reuses the prior entry value (no redundant lookup/history). - **55 lib tests pass**, including the `storage_changes_*` proptests, the Boojum storage-log trace tests, and a new `merged_storage_write_tracks_paid_and_rolls_back` covering non-zero `prepaid` + rollback. ## Measured impact On the eravm guest, the worst-case production batch (67912) drops from needing ~920–952 MiB to **fitting at ~720 MiB** (~200 MiB off peak) — turning a <32 MiB margin into comfortable headroom. ## Review items addressed P2 (re-export `StorageWriteEntry`), P3 (single insert in `write_storage`; add non-zero-paid test), and stale doc-comment field-name refs — all in `1044b47`. P1 (scope) addressed by this description; the API break is called out above for the coordinated eravm change. --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Reduces
WorldDiffmemory on large mainnet batches (consumer: matter-labs/eravm-airbender-verifier#18) by removing duplicated(address, key)keys across its maps. Stacked on #104 (vv/memopt-on-popzxc).Changes
read_storage_slots,written_storage_slots,committed_reads_at_depth_zero) into oneslot_flags: RollbackableMap<(H160,U256), u8>of bit flags. External-rollback semantics preserved; publiccommitted_reads_at_depth_zero_iterkept (now filters the flag).storage_changes: U256+paid_changes: u32) into onestorage_writes: RollbackableMap<(H160,U256), StorageWriteEntry { value, paid }>.transient_storage_changesleft separate (distinct keyspace).WorldDiff::get_storage_state()now returns&BTreeMap<_, StorageWriteEntry>(was…, U256>), andStorageWriteEntryis re-exported from the crate root. Direct callers must project.value. The downstream consumer is eravm-airbender-verifier'svm_fast(2 sites) — its vm2 pin bump must land together with that.valueprojection.Correctness
slot_flagsexternal;storage_writesinternal).write_storagedoes a single insert per path;prepaidreuses the prior entry value (no redundant lookup/history).storage_changes_*proptests, the Boojum storage-log trace tests, and a newmerged_storage_write_tracks_paid_and_rolls_backcovering non-zeroprepaid+ rollback.Measured impact
On the eravm guest, the worst-case production batch (67912) drops from needing ~920–952 MiB to fitting at ~720 MiB (~200 MiB off peak) — turning a <32 MiB margin into comfortable headroom.
Review items addressed
P2 (re-export
StorageWriteEntry), P3 (single insert inwrite_storage; add non-zero-paid test), and stale doc-comment field-name refs — all in1044b47. P1 (scope) addressed by this description; the API break is called out above for the coordinated eravm change.