Missed proposal getExecutionPayload: newExecutionPayload invalid

[leontodd]

Describe the bug
I've just missed a proposal on mainnet with the following Nimbus logs:

INF 2022-09-07 xx:xx:xx.xxx+xx:xx getExecutionPayload: newExecutionPayload invalid topics="beacval" executionPayloadStatus=ok(INVALID_BLOCK_HASH)
WRN 2022-09-07 xx:xx:xx.xxx+xx:xx Unable to get execution payload. Skipping block proposal topics="beacval" slot=xxxxxxx validator_index=xxxxxx

Geth showed this in its logs at the same time:

WARN [09-07|xx:xx:xx.xxx] Forkchoice requested update to zero hash

Mev-boost didn't show any logs at the time of proposal, but the closest lines were this:

time="2022-09-07Txx:xx:xx+xx:xx" level=info msg="http: GET /eth/v1/builder/status 200" duration=0.2572213 method=GET module=service path=/eth/v1/builder/status status=200
time="2022-09-07Txx:xx:xx+xx:xx" level=info msg="http: POST /eth/v1/builder/validators 200" duration=0.128918712 method=POST module=service path=/eth/v1/builder/validators status=200

The connection between the EL and CL seems to be fine as I have both Established connection to execution layer and Obtained terminal block from Engine API in my logs on startup.

To be honest I didn't know mev-boost even did anything yet until the merge!

To Reproduce
Steps to reproduce the behavior:

1. Platform details (OS, architecture): Ubuntu 20.04.5 LTS ARM64
2. Branch/commit used: v22.8.2

[TheAviationDoctor]

Had the exact same issue today of a block proposal failure on mainnet running Nimbus v22.8.2-3a8abd:

WRN 2022-09-08 xx:xx:xx.xxx Unable to get execution payload. Skipping block proposal topics="beacval" slot=xxxxxx validator_index=xxxxxx

At that point Besu v22.7.2 started throwing errors:

2022-09-08 xx:xx:xx.xxx | EthScheduler-Timer-0 | WARN  | BackwardSyncContext | There was an uncaught exception during Backwards Sync. Retrying in 5000 milliseconds...
java.util.concurrent.CompletionException: org.hyperledger.besu.ethereum.eth.manager.exceptions.MaxRetriesReachedException: Task failed: MAX_RETRIES_REACHED
        at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:331)
        at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:346)
        at java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:632)
        at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
        at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088)
        at org.hyperledger.besu.util.FutureUtils.lambda$propagateResult$1(FutureUtils.java:72)
        at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859)
        at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837)
        at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
        at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088)
        at org.hyperledger.besu.ethereum.eth.manager.task.AbstractRetryingPeerTask.executeTask(AbstractRetryingPeerTask.java:85)
        at org.hyperledger.besu.ethereum.eth.manager.task.AbstractEthTask.executeTaskTimed(AbstractEthTask.java:150)
        at org.hyperledger.besu.ethereum.eth.manager.EthScheduler.lambda$scheduleFutureTask$5(EthScheduler.java:159)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.hyperledger.besu.ethereum.eth.manager.exceptions.MaxRetriesReachedException: Task failed: MAX_RETRIES_REACHED
        ... 9 more

I had mev-boost active (main Flashbots relay) but no issue reported in its logs:

time="2022-09-08Txx:xx:xx" level=info msg="http: GET /eth/v1/builder/status 200" duration=0.512534505 method=GET module=service path=/eth/v1/builder/status status=200
time="2022-09-08Txx:xx:xx" level=info msg="http: POST /eth/v1/builder/validators 200" duration=0.263380585 method=POST module=service path=/eth/v1/builder/validators status=200
# BLOCK PROPOSAL FAILED HERE
time="2022-09-08Txx:xx:xx" level=info msg="http: GET /eth/v1/builder/status 200" duration=0.530426196 method=GET module=service path=/eth/v1/builder/status status=200
time="2022-09-08Txx:xx:xx" level=info msg="http: POST /eth/v1/builder/validators 200" duration=0.26355723 method=POST module=service path=/eth/v1/builder/validators status=200

Running Ubuntu 22.04.1 LTS AMD64.