Bws internal error fix

[jframe]

PR description

Fixes BWS issue where we are unable to import blocks for some hashes. We are currently getting this log at the end of the BWS.

{"@timestamp":"2024-01-31T10:58:00,982","level":"WARN","thread":"ForkJoinPool.commonPool-worker-374","class":"MergeCoordinator","message":"Internal error, backward sync completed but failed to import finalized block 0xd12ec91829da5af65ba80696d035bac054b8aac7cfcf60c983c12a5bcb1b926e","throwable":""}

This is due to adding hashes before have finished the initial sync and are ready to do a BWS.

Also moved the log message "Appending new head block hash {} to backward sync" into BackwardSyncContext so it is only logged if we are going to add the new block hash to the backward as may not kick off the BWS due to having not finished the initial sync.

Fixed Issue(s)

fixes #6527

Thanks for sending a pull request! Have you done the following?

• Checked out our contribution guidelines?
• Considered documentation and added the doc-change-required label to this PR if updates are required.
• Considered the changelog and included an update if required.
• For database changes (e.g. KeyValueSegmentIdentifier) considered compatibility and performed forwards and backwards compatibility tests

Locally, you can run these tests to catch failures early:

• unit tests: ./gradlew build
• acceptance tests: ./gradlew acceptanceTest
• integration tests: ./gradlew integrationTest
• reference tests: ./gradlew ethereum:referenceTests:referenceTests

[siladu]

Was adding these hashes during sync providing any benefit? (if it was, it was probably before this change... ed1480b)

When and how Besu receives FCU either during sync or during node recovery (e.g. after some besu downtime) varies between every CL.
Has sync and recovery been tested for each CL?
I think holesky would suffice if you haven't and decided to.

Happy to be convinced it doesn't need all this testing if you're confident it won't break anything :)

[jframe]

Was adding these hashes during sync providing any benefit? (if it was, it was probably before this change... ed1480b)

Don't think it was. Can't think of any benefit as the hashes would be out of date before we finished the sync.

When and how Besu receives FCU either during sync or during node recovery (e.g. after some besu downtime) varies between every CL. Has sync and recovery been tested for each CL? I think holesky would suffice if you haven't and decided to.

Haven't done that. That's a good idea. Will tests some syncs with different CLs on holesky.

Happy to be convinced it doesn't need all this testing if you're confident it won't break anything :)

[jframe]

Was adding these hashes during sync providing any benefit? (if it was, it was probably before this change... ed1480b)

Don't think it was. Can't think of any benefit as the hashes would be out of date before we finished the sync.

When and how Besu receives FCU either during sync or during node recovery (e.g. after some besu downtime) varies between every CL. Has sync and recovery been tested for each CL? I think holesky would suffice if you haven't and decided to.

Haven't done that. That's a good idea. Will tests some syncs with different CLs on holesky.

Happy to be convinced it doesn't need all this testing if you're confident it won't break anything :)

Tested Besu/Lighthouse, Besu/Lodestar, Besu/Nimbus, Besu/Prysm and Besu/Teku on holeksy