Hive test failure due to voluntary exits not being included after Deneb

[marioevz]

Describe the bug

We found an issue on a hive test regarding voluntary exits that seems to only be happening in Lodestar, and it seems like lodestar does not want to include them in a block.

The issue only reproduces if the test starts in Capella, waits for the Deneb fork, and then sends the voluntary exits.

In both configurations we have two lodestar clients with all validators attached, and we see similar messages to this:

Jan-09 15:32:21.955[rest]            �[34mdebug�[39m: Req req-1k 172.19.0.1 submitPoolVoluntaryExit
Jan-09 15:32:22.061[network]       �[36mverbose�[39m: Publish to topic topic=/eth2/383113c7/voluntary_exit/ssz_snappy, sentPeers=1

After submitting the (13) voluntary exits.

On Deneb genesis, the next produced block contains the exits:

Jan-09 15:32:27.017[chain]         �[36mverbose�[39m: Produced beacon block body fork=deneb, blockType=Full, slot=3, attestations=1, deposits=0, voluntaryExits=13, attesterSlashings=0, proposerSlashings=0, feeRecipientType=requested, feeRecipient=0x0000000000000000000000000000000000000000, executionPayloadPrepType=Cached, transactions=40, blobs=0, blsToExecutionChanges=0, withdrawals=16, executionPayloadValue=840000000000000

But on Capella genesis, the very next produced block (and all subsequent blocks) do not have any exits:

Jan-09 18:10:26.011[chain]         �[36mverbose�[39m: Produced beacon block body fork=deneb, blockType=Full, slot=35, attestations=1, deposits=0, voluntaryExits=0, attesterSlashings=0, proposerSlashings=0, feeRecipientType=requested, feeRecipient=0x0000000000000000000000000000000000000000, executionPayloadPrepType=Cached, transactions=40, blobs=0, blsToExecutionChanges=0, withdrawals=16, executionPayloadValue=840000000000000

Expected behavior

Voluntary exits must be included.

Steps to reproduce

git clone -b eth2-dencun https://github.com/marioevz/hive.git
cd hive
go build -v .
./hive --client-file ./configs/cancun.yaml --client nethermind,lodestar-bn,lodestar-vc --sim eth2/dencun --sim.limit "eth2-deneb-testnet/test-deneb-fork" --sim.loglevel 5

Test will fail because it times out waiting on validators to be set an exit epoch (which happens on all other clients).

Additional context

Log files here:
lodestar_voluntary_exits_logs.tar.gz

Same test is passing on other clients, and we are signing the voluntary exits using Capella domain.

One thing to consider is we run using unstable-linux-amd64, but in the log files I cannot fully confirm that this is the branch used to compile because the line shows empty:

Jan-09 20:30:43.875[]                 �[32minfo�[39m: Lodestar network=mainnet, version=v1.13.0, commit=

Operating system

Linux

Lodestar version or commit hash

e51f495

[dapplion]

Thank you so much for the super report! @marioevz ❤️

• I think I found the root cause in fix: update voluntary exit block inclusion filter #6278 will ask @nflaig for help double checking

[nflaig]

Thanks @marioevz for the detailed instructions on how to reproduce this 🙏

I was able to confirm that hive tests are passing using nflaig/lodestar:fix-exit-inclusion which includes the fix from @dapplion's branch (#6278).

Hive output

INFO[01-10|13:48:59] API: test ended                          suite=0 test=1 pass=true
INFO[01-10|13:48:59] API: suite ended                         suite=0
INFO[01-10|13:49:01] simulation eth2/dencun finished          suites=1 tests=1 failed=0

All submitted voluntary exists (13)

Jan-10 12:45:19.904[rest]            debug: Req req-g7 172.17.0.3 submitPoolVoluntaryExit

were included in the next block (slot=34, fork=deneb, voluntaryExits=13)

Jan-10 12:45:22.013[chain]         verbose: Produced beacon block body fork=deneb, blockType=Full, slot=34, attestations=1, deposits=0, voluntaryExits=13, attesterSlashings=0, proposerSlashings=0, feeRecipientType=requested, feeRecipient=0x0000000000000000000000000000000000000000, executionPayloadPrepType=Cached, transactions=40, shouldOverrideBuilder=false, blobs=0, blsToExecutionChanges=0, withdrawals=16, executionPayloadValue=840000000000000

[nflaig]

One thing to consider is we run using unstable-linux-amd64, but in the log files I cannot fully confirm that this is the branch used to compile because the line shows empty:

The might be due to how the image is build in eth-client-docker-image-builder. To be able to get access to the commit hash we need to pass it as build arg which we do in our CI but I don't see this in their workflows.

lodestar/.github/workflows/publish-stable.yml

Line 154 in aca6354

--build-arg COMMIT=$(git rev-parse HEAD)

Should we open an issue on the ethpandaops repo?

[marioevz]

Should we open an issue on the ethpandaops repo?

Yes please! It's not a breaking issue, but it's nice to be sure what branch we are running on.