Improve pending blocks retrieval mechanism #4227

Gabriel-Trintinalia · 2022-08-08T03:21:25Z

Signed-off-by: Gabriel Trintinalia gabriel.trintinalia@consensys.net

PR description

When a block is saved for future import, requesting the lowest announced block parent does not seem to be always the best option. It can be that we cannot retrieve this block and then we keep selecting the lowest block over and over and eventually it does not fix the gap. A better approach for trying to unstick could be to request the parent of the lowest pending ancestor

Changes:

When saving a block for future import, try to retrieve the parent of the lowest pending block connected to this block.
When saving a block for future import, do not try to retrieve the parent the lowest announced block.
Avoid requesting the same parent block several times when saving pending blocks
Improve logs when saving/importing a block

Fixed Issue(s)

potential aid for #3955
#4197

Documentation

I thought about documentation and added the doc-change-required label to this PR if
updates are required.

Changelog

I thought about the changelog and included a changelog update if required.

Signed-off-by: Gabriel Trintinalia <gabriel.trintinalia@consensys.net>

ethereum/core/src/main/java/org/hyperledger/besu/ethereum/core/BlockHeader.java

ethereum/eth/src/main/java/org/hyperledger/besu/ethereum/eth/sync/BlockPropagationManager.java

...eum/eth/src/main/java/org/hyperledger/besu/ethereum/eth/sync/state/PendingBlocksManager.java

ethereum/eth/src/main/java/org/hyperledger/besu/ethereum/eth/sync/tasks/PersistBlockTask.java

...rc/test/java/org/hyperledger/besu/ethereum/eth/sync/AbstractBlockPropagationManagerTest.java

Signed-off-by: Gabriel Trintinalia <gabriel.trintinalia@consensys.net>

…-Trintinalia/besu into 3955-besu-stop-importing

Signed-off-by: Gabriel Trintinalia <gabriel.trintinalia@consensys.net>

…ancestors of Block Signed-off-by: Gabriel Trintinalia <gabriel.trintinalia@consensys.net>

fab-10

This goes in the right directions, implementing a basic backward sync to fill gaps in the block propagation, I have reported just a couple of minor things.

During this code review, I have also reviewed the role of maybeProcessNonAnnouncedBlocks that looks forward, so in the future to try to proactively fetch new blocks, that usually should be notified by peers soon after, so not sure how much it is useful, instead this class was missing something that looks backward, that is probably much needed, not sure if the author relied on the fact that full sync eventally fills the gaps, but when full sync stalls (for yet unknown reasons) we are in throuble, and this PR should fix that!

...eum/eth/src/main/java/org/hyperledger/besu/ethereum/eth/sync/state/PendingBlocksManager.java

ethereum/eth/src/main/java/org/hyperledger/besu/ethereum/eth/sync/BlockPropagationManager.java

Signed-off-by: Gabriel Trintinalia <gabriel.trintinalia@consensys.net>

…ing pending block to the list Signed-off-by: Gabriel Trintinalia <gabriel.trintinalia@consensys.net>

Signed-off-by: Gabriel Trintinalia <gabriel.trintinalia@consensys.net>

…block Signed-off-by: Gabriel Trintinalia <gabriel.trintinalia@consensys.net>

ajsutton

I'm leaning on @fab-10 's review here a fair bit for the finer details of how this works but looks good to my understanding.

ethereum/eth/src/main/java/org/hyperledger/besu/ethereum/eth/sync/BlockPropagationManager.java

...eth/src/test/java/org/hyperledger/besu/ethereum/eth/sync/state/PendingBlocksManagerTest.java

pinges

LGTM, you might want to think about getting rid of the second parameter of the pendingAncestorBlockOf() method.
Some finals missing :-)

pinges · 2022-08-12T01:38:25Z

ethereum/eth/src/main/java/org/hyperledger/besu/ethereum/eth/sync/BlockPropagationManager.java

+                  LOG.info(
+                      "Imported {} pending blocks: {}",
+                      r.size(),
+                      r.stream().map(b -> b.getHeader().getNumber()).collect(Collectors.toList()));
                }
                if (t != null) {
                  LOG.error("Error importing pending blocks", t);


What happens if there was an error? Do we need to remove a block from the pending blocks? Depending on what error was thrown?

ethereum/eth/src/main/java/org/hyperledger/besu/ethereum/eth/sync/BlockPropagationManager.java

Signed-off-by: Gabriel Trintinalia <gabriel.trintinalia@consensys.net>

* Add more log to retrieve parent method * Request the lowest pending ancestor when saving a block * Replace recursive implementation with iterative when getting pending ancestors of Block * Decrease scope of synchronized block to reflect only the event of adding pending block to the list * Add fork to the chain so test is more representative Signed-off-by: Gabriel Trintinalia <gabriel.trintinalia@consensys.net> Signed-off-by: Gabriel Trintinalia <gabriel.trintinalia@consensys.net> Co-authored-by: Gabriel Trintinalia <gabriel.trintinalia@consensys.net>

Add more log to retrieve parent method

97f21b7

Signed-off-by: Gabriel Trintinalia <gabriel.trintinalia@consensys.net>

Gabriel-Trintinalia marked this pull request as draft August 8, 2022 03:22

Fix spotless

149ff69

Signed-off-by: Gabriel Trintinalia <gabriel.trintinalia@consensys.net>

non-fungible-nelson added the TeamRevenant GH issues worked on by Revenant Team label Aug 8, 2022

macfarla added the mainnet label Aug 9, 2022

Gabriel-Trintinalia and others added 5 commits August 9, 2022 16:26

Request the lowest pending ancestor when saving a block

46c7754

Signed-off-by: Gabriel Trintinalia <gabriel.trintinalia@consensys.net>

Add tests

249dd87

Signed-off-by: Gabriel Trintinalia <gabriel.trintinalia@consensys.net>

Merge branch 'main' into 3955-besu-stop-importing

d6232c7

Remove cache exposure

ffbf5be

Signed-off-by: Gabriel Trintinalia <gabriel.trintinalia@consensys.net>

Remove unused method

ef44e09

Signed-off-by: Gabriel Trintinalia <gabriel.trintinalia@consensys.net>

Gabriel-Trintinalia mentioned this pull request Aug 9, 2022

Besu stops importing new blocks on mainnet #3955

Closed

Gabriel-Trintinalia changed the title ~~WIP: Add more log to retrieve parent method~~ WIP: Request the parent of the lowest pending block instead of lowest announced block Aug 9, 2022

Gabriel-Trintinalia and others added 3 commits August 10, 2022 08:49

Merge branch 'main' into 3955-besu-stop-importing

92bcd4b

Remove double space

e6fa24a

Signed-off-by: Gabriel Trintinalia <gabriel.trintinalia@consensys.net>

Add method description

d8bddcb

Signed-off-by: Gabriel Trintinalia <gabriel.trintinalia@consensys.net>

Gabriel-Trintinalia changed the title ~~WIP: Request the parent of the lowest pending block instead of lowest announced block~~ Improve pending blocks retrieval mechanism Aug 9, 2022

Merge branch 'main' into 3955-besu-stop-importing

bb9a127

Gabriel-Trintinalia marked this pull request as ready for review August 9, 2022 23:42

Merge branch 'main' into 3955-besu-stop-importing

dd54e83

ajsutton reviewed Aug 10, 2022

View reviewed changes

Gabriel-Trintinalia added 4 commits August 10, 2022 13:40

Rollback log message changes

77b26c3

Signed-off-by: Gabriel Trintinalia <gabriel.trintinalia@consensys.net>

Merge branch '3955-besu-stop-importing' of https://github.com/Gabriel…

c55787d

…-Trintinalia/besu into 3955-besu-stop-importing

Change log level when parent is already requested

887a43d

Signed-off-by: Gabriel Trintinalia <gabriel.trintinalia@consensys.net>

Replace recursive implementation with iterative when getting pending …

03ba11d

…ancestors of Block Signed-off-by: Gabriel Trintinalia <gabriel.trintinalia@consensys.net>

fab-10 requested changes Aug 10, 2022

View reviewed changes

...eum/eth/src/main/java/org/hyperledger/besu/ethereum/eth/sync/state/PendingBlocksManager.java Outdated Show resolved Hide resolved

ethereum/eth/src/main/java/org/hyperledger/besu/ethereum/eth/sync/BlockPropagationManager.java Outdated Show resolved Hide resolved

Gabriel-Trintinalia and others added 5 commits August 11, 2022 11:53

Merge branch 'hyperledger:main' into 3955-besu-stop-importing

39ea999

Remove null check

54615a9

Signed-off-by: Gabriel Trintinalia <gabriel.trintinalia@consensys.net>

Decrease scope of synchronized block to reflect only the event of add…

53f0654

…ing pending block to the list Signed-off-by: Gabriel Trintinalia <gabriel.trintinalia@consensys.net>

Simplify savePendingBlock method

1283019

Signed-off-by: Gabriel Trintinalia <gabriel.trintinalia@consensys.net>

Remove local variable

0a16734

Signed-off-by: Gabriel Trintinalia <gabriel.trintinalia@consensys.net>

Gabriel-Trintinalia added 2 commits August 11, 2022 19:10

Refine tests

202a079

Signed-off-by: Gabriel Trintinalia <gabriel.trintinalia@consensys.net>

Refine tests

e5c1403

Signed-off-by: Gabriel Trintinalia <gabriel.trintinalia@consensys.net>

fab-10 approved these changes Aug 11, 2022

View reviewed changes

Add maxAncestorBlock parameter when searching for a pending ancestor …

693ebae

…block Signed-off-by: Gabriel Trintinalia <gabriel.trintinalia@consensys.net>

ajsutton approved these changes Aug 12, 2022

View reviewed changes

ethereum/eth/src/main/java/org/hyperledger/besu/ethereum/eth/sync/BlockPropagationManager.java Show resolved Hide resolved

...eth/src/test/java/org/hyperledger/besu/ethereum/eth/sync/state/PendingBlocksManagerTest.java Outdated Show resolved Hide resolved

pinges approved these changes Aug 12, 2022

View reviewed changes

Gabriel-Trintinalia added 3 commits August 12, 2022 12:52

Add final to local variables

17e45dd

Signed-off-by: Gabriel Trintinalia <gabriel.trintinalia@consensys.net>

Remove maxAncestorLevel parameter

8e91997

Signed-off-by: Gabriel Trintinalia <gabriel.trintinalia@consensys.net>

Add fork to the chain so test is more representative

1bae5c5

Signed-off-by: Gabriel Trintinalia <gabriel.trintinalia@consensys.net>

macfarla enabled auto-merge (squash) August 12, 2022 03:54

Change CHANGELOG.md

cdc6dd1

Signed-off-by: Gabriel Trintinalia <gabriel.trintinalia@consensys.net>

auto-merge was automatically disabled August 12, 2022 03:59
Head branch was pushed to by a user without write access

macfarla merged commit f847ead into hyperledger:main Aug 12, 2022

Gabriel-Trintinalia deleted the 3955-besu-stop-importing branch October 5, 2022 01:30

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improve pending blocks retrieval mechanism #4227

Improve pending blocks retrieval mechanism #4227

Gabriel-Trintinalia commented Aug 8, 2022 •

edited

Loading

fab-10 left a comment

ajsutton left a comment

pinges left a comment

pinges Aug 12, 2022

Improve pending blocks retrieval mechanism #4227

Improve pending blocks retrieval mechanism #4227

Conversation

Gabriel-Trintinalia commented Aug 8, 2022 • edited Loading

PR description

Fixed Issue(s)

Documentation

Changelog

fab-10 left a comment

Choose a reason for hiding this comment

ajsutton left a comment

Choose a reason for hiding this comment

pinges left a comment

Choose a reason for hiding this comment

pinges Aug 12, 2022

Choose a reason for hiding this comment

Gabriel-Trintinalia commented Aug 8, 2022 •

edited

Loading