Snapsync persist state #4381

matkt · 2022-09-12T15:06:49Z

PR description

This PR avoids starting the download of the world state from scratch when restarting Besu

Fixed Issue(s)

Documentation

I thought about documentation and added the doc-change-required label to this PR if
updates are required.

Changelog

I thought about the changelog and included a changelog update if required.

Signed-off-by: Karim TAAM <karim.t2am@gmail.com>

services/tasks/src/main/java/org/hyperledger/besu/services/tasks/FlatFileTaskCollection.java

Signed-off-by: Karim TAAM <karim.t2am@gmail.com>

…re/snapsync-persist Signed-off-by: Karim TAAM <karim.t2am@gmail.com>

Signed-off-by: Karim TAAM <karim.t2am@gmail.com>

garyschulte

intuitively I think that keeping the pending and inconsistent ranges in files in the filesystem rather than in rocksdb would be more flexible than adding column families. It is clearly a lot more data, but it would be consistent with how we treat pivot block.

Are we expecting that the number of files would be just too unwieldy?

garyschulte · 2022-10-24T20:59:09Z

.../src/main/java/org/hyperledger/besu/ethereum/storage/keyvalue/KeyValueSegmentIdentifier.java

@@ -33,7 +33,9 @@ public enum KeyValueSegmentIdentifier implements SegmentIdentifier {
  GOQUORUM_PRIVATE_STORAGE(new byte[] {12}),
  BACKWARD_SYNC_HEADERS(new byte[] {13}),
  BACKWARD_SYNC_BLOCKS(new byte[] {14}),
-  BACKWARD_SYNC_CHAIN(new byte[] {15});
+  BACKWARD_SYNC_CHAIN(new byte[] {15}),
+  SNAPSYNC_MISSING_ACCOUNT_RANGE(new byte[] {16}),


we should mark this PR as breaking backward compatibility if we add these column families

garyschulte · 2022-10-24T20:59:49Z

.../src/main/java/org/hyperledger/besu/ethereum/storage/keyvalue/WorldStateKeyValueStorage.java

@@ -30,6 +30,7 @@
 import java.util.function.Predicate;
 import java.util.stream.Stream;

+import kotlin.Pair;


can we use java 17 record types yet?

I propose to use org.apache.commons.lang3.tuple.Pair; is it fine for you ?

because we are using java 11

apache commons is good by me. We are so close to being able to use record types though... 🤤

garyschulte · 2022-10-24T21:05:00Z

.../src/main/java/org/hyperledger/besu/ethereum/storage/keyvalue/WorldStateKeyValueStorage.java

+    try (final Stream<Pair<byte[], byte[]>> entry = keyValueStorage.stream()) {
+      entry.forEach(
          key -> {
            lock.lock();
            try {
-              if (!inUseCheck.test(key) && keyValueStorage.tryDelete(key)) {
+              if (!inUseCheck.test(key.getFirst()) && keyValueStorage.tryDelete(key.getFirst())) {


could we not use streamKeys() ? It looks like we are only using Pair.getFirst() here and in other places. I suspect the iterator for just keys would be more performant in cases where we only want keys 🤔

garyschulte · 2022-10-24T21:09:03Z

...org/hyperledger/besu/plugin/services/storage/rocksdb/unsegmented/RocksDBKeyValueStorage.java

+  public Stream<Pair<byte[], byte[]>> stream() {
    final RocksIterator rocksIterator = db.newIterator();
    rocksIterator.seekToFirst();
-    return RocksDbKeyIterator.create(rocksIterator).toStream();
+    return RocksDbIterator.create(rocksIterator).toStream();
+  }


seems like we could offer both stream() and streamKeys()

garyschulte · 2022-10-25T01:10:56Z

Are we expecting that the number of files would be just too unwieldy?

Maybe rocksdb transactions are just cleaner to deal with than files ? 🤔

matkt · 2022-10-25T06:51:11Z

Are we expecting that the number of files would be just too unwieldy?

Maybe rocksdb transactions are just cleaner to deal with than files ? 🤔

I tried a lot at the beginning to go through a file and I never managed to get a something clean. So I decided to switch to rocksdb .

Are we expecting that the number of files would be just too unwieldy?

Maybe rocksdb transactions are just cleaner to deal with than files ? 🤔

Yes, the number of accounts to fix is something that cannot really be known over time. The bigger the chain, the bigger this list will be. And so you need a mechanism to manage this increase. I tried a lot to go through a file that splits when the size reaches a limit but after I said that I try to redo what rocksdb does better. So I decided to use rocksdb. But if you have a better idea to go through a file that could be split via a library why not

Signed-off-by: Karim TAAM <karim.t2am@gmail.com>

garyschulte

LGTM. The only open question is whether or not the new column families break backward compatibility. @gezero did you make besu handle extra column families or just bela?

Signed-off-by: Karim TAAM <karim.t2am@gmail.com>

This PR avoids restarting the download of the world state from scratch when restarting Besu Signed-off-by: Karim TAAM <karim.t2am@gmail.com> Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com>

This PR avoids restarting the download of the world state from scratch when restarting Besu Signed-off-by: Karim TAAM <karim.t2am@gmail.com>

persisted context for snapsync

52f66b1

Signed-off-by: Karim TAAM <karim.t2am@gmail.com>

matkt added snapsync mainnet labels Sep 12, 2022

github-advanced-security bot found potential problems Sep 12, 2022

View reviewed changes

services/tasks/src/main/java/org/hyperledger/besu/services/tasks/FlatFileTaskCollection.java Fixed Show fixed Hide fixed

matkt force-pushed the feature/snapsync-persist branch from 3985ca0 to fd8e70c Compare September 14, 2022 13:46

clean code

16e1833

Signed-off-by: Karim TAAM <karim.t2am@gmail.com>

matkt force-pushed the feature/snapsync-persist branch from fd8e70c to 16e1833 Compare September 14, 2022 14:07

matkt added 3 commits September 14, 2022 16:17

fix pipeline

f6e8737

Signed-off-by: Karim TAAM <karim.t2am@gmail.com>

fix key value storage issue and remove some useless logs

b2802d4

Signed-off-by: Karim TAAM <karim.t2am@gmail.com>

fix issue

26f0a74

Signed-off-by: Karim TAAM <karim.t2am@gmail.com>

matkt force-pushed the feature/snapsync-persist branch from c6fdb82 to 26f0a74 Compare September 15, 2022 14:44

Merge branch 'main' of https://github.com/hyperledger/besu into featu…

a2d7a1e

…re/snapsync-persist Signed-off-by: Karim TAAM <karim.t2am@gmail.com>

matkt marked this pull request as ready for review September 19, 2022 14:45

siladu and others added 2 commits September 26, 2022 15:28

Merge branch 'main' into feature/snapsync-persist

eb3f21d

Merge branch 'main' into feature/snapsync-persist

b7c82f3

Signed-off-by: Karim TAAM <karim.t2am@gmail.com>

garyschulte reviewed Oct 24, 2022

View reviewed changes

matkt added 5 commits October 26, 2022 10:00

fix review

949d2e9

Signed-off-by: Karim TAAM <karim.t2am@gmail.com>

fix build

e961e51

Signed-off-by: Karim TAAM <karim.t2am@gmail.com>

fix build

315e11d

Signed-off-by: Karim TAAM <karim.t2am@gmail.com>

update plugin api hash

0734074

Signed-off-by: Karim TAAM <karim.t2am@gmail.com>

missing header

ede3711

Signed-off-by: Karim TAAM <karim.t2am@gmail.com>

garyschulte approved these changes Oct 27, 2022

View reviewed changes

matkt added 2 commits November 1, 2022 13:51

Merge branch 'main' into feature/snapsync-persist

35684e8

update CHANGELOG

db2e5ad

Signed-off-by: Karim TAAM <karim.t2am@gmail.com>

matkt enabled auto-merge (squash) November 1, 2022 12:55

matkt merged commit da9b107 into hyperledger:main Nov 1, 2022

eum602 pushed a commit to lacchain/besu that referenced this pull request Nov 3, 2023

Snapsync persist state (hyperledger#4381)

68ff109

This PR avoids restarting the download of the world state from scratch when restarting Besu Signed-off-by: Karim TAAM <karim.t2am@gmail.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Snapsync persist state #4381

Snapsync persist state #4381

matkt commented Sep 12, 2022 •

edited

Loading

garyschulte left a comment •

edited

Loading

garyschulte Oct 24, 2022

garyschulte Oct 24, 2022

matkt Oct 26, 2022

matkt Oct 26, 2022

garyschulte Oct 27, 2022

garyschulte Oct 24, 2022 •

edited

Loading

matkt Oct 25, 2022

matkt Oct 26, 2022

garyschulte Oct 24, 2022

matkt Oct 25, 2022

matkt Oct 26, 2022

garyschulte commented Oct 25, 2022

matkt commented Oct 25, 2022

garyschulte left a comment

Snapsync persist state #4381

Snapsync persist state #4381

Conversation

matkt commented Sep 12, 2022 • edited Loading

PR description

Fixed Issue(s)

Documentation

Changelog

garyschulte left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

garyschulte Oct 24, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

garyschulte commented Oct 25, 2022

matkt commented Oct 25, 2022

garyschulte left a comment

Choose a reason for hiding this comment

matkt commented Sep 12, 2022 •

edited

Loading

garyschulte left a comment •

edited

Loading

garyschulte Oct 24, 2022 •

edited

Loading