Bonsai snapshot worldstate #4409

garyschulte · 2022-09-16T21:48:20Z

PR description

This PR adds support for using rocksdb snapshot transactions as the basis for a bonsai worldstate. This gives us the ability to have an isolated copy of the world state for concurrent operations like block production, block validation, transaction validation, historical state queries, etc.

The notion is to merge this capability into main, and start replacing the uses of BonsaiLayeredWorldState and direct uses of BonsaiInMemoryWorldState when we need isolated and/or concurrent operations on the worldstate.

This is a strictly additive PR and does not change existing implementations. Uses of snapshot worldstates will be useful to resolve issues such as #4372 #4250 #4199 #4151 and other bonsai concurrency issues.

Changes in support of snapshot worldstate include:

moving (back) to OptimisticTransactionDB, so we can mutate the snapshot transactions
creating BonsaiWorldStateKeyValueStorage.BonsaiUpdater interface extending WorldStateStorage.Updater, so we can have a snapshot specific Updater implementation
Addition of SnappableStorage, SnappedStorage to plugin-api in order to support/expose the snapshot behavior

Fixed Issue(s)

relates to #4402
relates to #4403

Documentation

I thought about documentation and added the doc-change-required label to this PR if
updates are required.

Changelog

I thought about the changelog and included a changelog update if required.

matkt · 2022-09-17T12:15:32Z

Excellent work I will review it on Monday or this weekend

matkt

I will continue the review but i already share this

config/src/main/java/org/hyperledger/besu/config/GenesisAllocation.java

matkt · 2022-09-17T18:50:43Z

.../main/java/org/hyperledger/besu/ethereum/bonsai/BonsaiSnapshotWorldStateKeyValueStorage.java

+  }
+
+  public static class SnapshotUpdater implements BonsaiWorldStateKeyValueStorage.BonsaiUpdater {
+    //    private static final Logger LOG =


I think you missed this line

matkt

Seems to be ok. waiting for the next PR related to snapshot

garyschulte · 2022-10-13T14:44:40Z

IMO, this is ready to merge to main to clear the way for subsequent PRs that implement bonsai snapshots.

gezero · 2022-10-13T15:02:40Z

ethereum/core/src/main/java/org/hyperledger/besu/ethereum/bonsai/BonsaiSnapshotWorldState.java

+    this.trieBranchSnap = (SnappedKeyValueStorage) snapshotWorldStateStorage.trieBranchStorage;
+  }
+
+  public static BonsaiSnapshotWorldState create(


Is there a potential race with the storages being changed while the snapshots are generated?

There is some risk of that theoretically, but considering we write to rocksdb in batch commits and the snapshots are essentially atomic, this is probably very very minimal chance in practice. There is certainly far less concurrency risk than our current strategy where the underlying persisted state can change underneath a BonsaiInMemoryWorldState or BonsaiLayeredWorldState

It seems reasonable to implement locking in BonsaiPersistedWorldState so that we do not commit a batch write (via calculateRootHash) and take a snapshot concurrently. It should be a very short lock for getting a snapshot, and delaying getting a snapshot while waiting on a lock to release from a batch write would normally not be significant.

I will add this in one of the subsequent snapshot PRs and add tests 👍

gezero · 2022-10-13T15:13:54Z

...g/hyperledger/besu/plugin/services/storage/rocksdb/segmented/RocksDBSnapshotTransaction.java

+    } catch (final RocksDBException e) {
+      if (e.getMessage().contains(NO_SPACE_LEFT_ON_DEVICE)) {
+        LOG.error(e.getMessage());
+        System.exit(0);


System.exit feels too brutal to me. Is there any chance this can corrupt db for instance in case an ongoing update is happening in parallel? Maybe we don't access db from different threads.

Good question. This is the same behavior as RocksDBTransaction. The right course of action isn't really clear since we do not have any disk space to work with.

Signed-off-by: garyschulte <garyschulte@gmail.com>

Isolation is working, but unclear whether trielogs are persisting correctly since we cannot create a snapshot and roll it to a particular hash in unit tests Signed-off-by: garyschulte <garyschulte@gmail.com>

snapshot trieLog is persistable snapshot roll forward/back is FAILING Signed-off-by: garyschulte <garyschulte@gmail.com>

Signed-off-by: garyschulte <garyschulte@gmail.com>

…correctly dispose of snapshots Signed-off-by: garyschulte <garyschulte@gmail.com>

Signed-off-by: garyschulte <garyschulte@gmail.com>

… extend persisted worldstate rather than in-memory worldstate Signed-off-by: garyschulte <garyschulte@gmail.com>

close snapshot worldstate transactions on close release snapshot on close of snapshot transaction (potentially fraught) Signed-off-by: garyschulte <garyschulte@gmail.com>

* use optimistictransactiondb for mutable isolated snapshots * plumbing necessary to have a snapshot specific updater. * snapshot rolling working * implement AutoCloseable on BonsaiSnapshotWorldState to ensure we can correctly dispose of snapshots * add snapshot transaction cloning, change snapshot based worldstate to extend persisted worldstate rather than in-memory worldstate Signed-off-by: garyschulte <garyschulte@gmail.com> Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com>

* use optimistictransactiondb for mutable isolated snapshots * plumbing necessary to have a snapshot specific updater. * snapshot rolling working * implement AutoCloseable on BonsaiSnapshotWorldState to ensure we can correctly dispose of snapshots * add snapshot transaction cloning, change snapshot based worldstate to extend persisted worldstate rather than in-memory worldstate Signed-off-by: garyschulte <garyschulte@gmail.com>

garyschulte requested review from matkt and shemnon September 16, 2022 21:54

garyschulte force-pushed the feature/snapshot-worldstate-reprise branch from 936df87 to d983043 Compare September 16, 2022 23:32

matkt reviewed Sep 17, 2022

View reviewed changes

matkt approved these changes Sep 19, 2022

View reviewed changes

totoCZ mentioned this pull request Sep 24, 2022

Teku + Besu, attestations randomly missed since the merge Consensys/teku#6205

Closed

garyschulte force-pushed the feature/snapshot-worldstate-reprise branch 2 times, most recently from d1edc6b to 12c045c Compare October 10, 2022 18:28

garyschulte marked this pull request as draft October 11, 2022 00:04

garyschulte force-pushed the feature/snapshot-worldstate-reprise branch from 01725fe to c675a28 Compare October 11, 2022 17:08

garyschulte marked this pull request as ready for review October 11, 2022 19:29

garyschulte marked this pull request as draft October 11, 2022 22:19

garyschulte force-pushed the feature/snapshot-worldstate-reprise branch from c675a28 to 158937a Compare October 12, 2022 21:36

garyschulte marked this pull request as ready for review October 13, 2022 14:41

garyschulte force-pushed the feature/snapshot-worldstate-reprise branch 2 times, most recently from 4fe232b to 04a0820 Compare October 13, 2022 15:04

gezero approved these changes Oct 13, 2022

View reviewed changes

garyschulte added 11 commits October 13, 2022 09:21

squash commits from bonsai isolation for easier rebase

dde8d65

Signed-off-by: garyschulte <garyschulte@gmail.com>

use optimistictransactiondb for mutable isolated snapshots

a8864b9

Signed-off-by: garyschulte <garyschulte@gmail.com>

plumbing necessary to have a snapshot specific updater.

ccf9bf8

Isolation is working, but unclear whether trielogs are persisting correctly since we cannot create a snapshot and roll it to a particular hash in unit tests Signed-off-by: garyschulte <garyschulte@gmail.com>

unit tests, snapshot isolation tests are passing

c2fce2f

snapshot trieLog is persistable snapshot roll forward/back is FAILING Signed-off-by: garyschulte <garyschulte@gmail.com>

trying to get snapshot log rolling working

897ee61

Signed-off-by: garyschulte <garyschulte@gmail.com>

snapshot rolling working

87768da

Signed-off-by: garyschulte <garyschulte@gmail.com>

comment clean up

9c6a2c4

Signed-off-by: garyschulte <garyschulte@gmail.com>

implement AutoCloseable on BonsaiSnapshotWorldState to ensure we can …

221bab3

…correctly dispose of snapshots Signed-off-by: garyschulte <garyschulte@gmail.com>

add state root checks to the snapshot isolation tests

b74ce5b

Signed-off-by: garyschulte <garyschulte@gmail.com>

spdx headers, test and comment cleanup

3791664

Signed-off-by: garyschulte <garyschulte@gmail.com>

fix mocks

ff99a8b

Signed-off-by: garyschulte <garyschulte@gmail.com>

garyschulte added 3 commits October 13, 2022 09:21

fix unit test

12ade42

Signed-off-by: garyschulte <garyschulte@gmail.com>

add snapshot transaction cloning, change snapshot based worldstate to…

16fe37c

… extend persisted worldstate rather than in-memory worldstate Signed-off-by: garyschulte <garyschulte@gmail.com>

change snapshot worldstate back to extending BonsaiInMemoryWorldState

947ae51

close snapshot worldstate transactions on close release snapshot on close of snapshot transaction (potentially fraught) Signed-off-by: garyschulte <garyschulte@gmail.com>

garyschulte force-pushed the feature/snapshot-worldstate-reprise branch from 04a0820 to 947ae51 Compare October 13, 2022 16:21

garyschulte enabled auto-merge (squash) October 13, 2022 16:28

garyschulte merged commit d73ce21 into hyperledger:main Oct 13, 2022

This was referenced Nov 5, 2022

World State Root does not match expected value -> Error processing block/Failed to import block: Invalid block #4151

Closed

Critical Exception Processing Transaction #4079

Closed

Critical Exception Processing Transaction #4250

Closed

garyschulte deleted the feature/snapshot-worldstate-reprise branch January 24, 2024 21:33

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bonsai snapshot worldstate #4409

Bonsai snapshot worldstate #4409

garyschulte commented Sep 16, 2022 •

edited

Loading

matkt commented Sep 17, 2022 •

edited

Loading

matkt left a comment

matkt Sep 17, 2022

matkt left a comment

garyschulte commented Oct 13, 2022

gezero Oct 13, 2022

garyschulte Oct 13, 2022

garyschulte Oct 13, 2022 •

edited

Loading

gezero Oct 13, 2022

garyschulte Oct 13, 2022

Bonsai snapshot worldstate #4409

Bonsai snapshot worldstate #4409

Conversation

garyschulte commented Sep 16, 2022 • edited Loading

PR description

Fixed Issue(s)

Documentation

Changelog

matkt commented Sep 17, 2022 • edited Loading

matkt left a comment

Choose a reason for hiding this comment

matkt Sep 17, 2022

Choose a reason for hiding this comment

matkt left a comment

Choose a reason for hiding this comment

garyschulte commented Oct 13, 2022

gezero Oct 13, 2022

Choose a reason for hiding this comment

garyschulte Oct 13, 2022

Choose a reason for hiding this comment

garyschulte Oct 13, 2022 • edited Loading

Choose a reason for hiding this comment

gezero Oct 13, 2022

Choose a reason for hiding this comment

garyschulte Oct 13, 2022

Choose a reason for hiding this comment

garyschulte commented Sep 16, 2022 •

edited

Loading

matkt commented Sep 17, 2022 •

edited

Loading

garyschulte Oct 13, 2022 •

edited

Loading