Vouch cannot attest via Lodestar

[yorickdowne]

Describe the bug

Our Vouch 1.7 shows the Lodestar CLs as not available for attestations. We haven't tracked this down as to which Lodestar version may have caused a change in behavior.

Please let us know what we can get you to narrow this down.

Expected behavior

Vouch works with Lodestar

Steps to reproduce

Configure Lodestar Ethereum node, and Vouch pointed at it and at least one other CL

Additional context

Logs from Lodestar:

consensus-1  | Feb-09 10:03:05.000[]                 info: Synced - slot: 8387413 - head: 0xcf20…9930 - exec-block: valid(19189850 0xe5d8…) - finalized: 0x15f9…93f2:262104 - peers: 22
consensus-1  | Feb-09 10:03:07.073[rest]            error: Req req-43 getAggregatedAttestation error - No attestation for slot=8387413 dataRoot=0xcec0c9b1f76ab7e46ab74f654122c44c90ee97aa1356ad9b6777e5df0871835b
consensus-1  | Error: No attestation for slot=8387413 dataRoot=0xcec0c9b1f76ab7e46ab74f654122c44c90ee97aa1356ad9b6777e5df0871835b
consensus-1  |     at AttestationPool.getAggregate (file:///usr/app/packages/beacon-node/src/chain/opPools/attestationPool.ts:136:13)
consensus-1  |     at Object.getAggregatedAttestation (file:///usr/app/packages/beacon-node/src/api/impl/validator/index.ts:1040:47)
consensus-1  |     at processTicksAndRejections (node:internal/process/task_queues:95:5)
consensus-1  |     at Object.handler (file:///usr/app/packages/api/src/utils/server/genericJsonServer.ts:45:23)
consensus-1  | Feb-09 10:03:17.020[]                 info: Synced - slot: 8387414 - head: 0x5d51…c772 - exec-block: valid(19189851 0xfe95…) - finalized: 0x15f9…93f2:262104 - peers: 25
consensus-1  | Feb-09 10:03:28.999[]                 info: Synced - slot: 8387415 - head: 0xeed8…ceb6 - exec-block: valid(19189852 0x2908…) - finalized: 0x15f9…93f2:262104 - peers: 24
consensus-1  | Feb-09 10:03:31.071[rest]            error: Req req-4n getAggregatedAttestation error - No attestation for slot=8387415 dataRoot=0x82c2090fec4d56f5176c183280f9d2f63a2ccf7b3ecb419510ef9e82c0f0a895
consensus-1  | Error: No attestation for slot=8387415 dataRoot=0x82c2090fec4d56f5176c183280f9d2f63a2ccf7b3ecb419510ef9e82c0f0a895
consensus-1  |     at AttestationPool.getAggregate (file:///usr/app/packages/beacon-node/src/chain/opPools/attestationPool.ts:136:13)
consensus-1  |     at Object.getAggregatedAttestation (file:///usr/app/packages/beacon-node/src/api/impl/validator/index.ts:1040:47)
consensus-1  |     at processTicksAndRejections (node:internal/process/task_queues:95:5)
consensus-1  |     at Object.handler (file:///usr/app/packages/api/src/utils/server/genericJsonServer.ts:45:23)
consensus-1  | Feb-09 10:03:41.079[]                 info: Synced - slot: 8387416 - head: 0xd216…9251 - exec-block: valid(19189853 0xb04b…) - finalized: 0x15f9…93f2:262104 - peers: 37
consensus-1  | Feb-09 10:03:43.045[rest]            error: Req req-4y getAggregatedAttestation error - No attestation for slot=8387416 dataRoot=0xb7ec7ae44e848229f5431c67e7e9305b87c03654f43a1de3d9cc1c075d9f223d
consensus-1  | Error: No attestation for slot=8387416 dataRoot=0xb7ec7ae44e848229f5431c67e7e9305b87c03654f43a1de3d9cc1c075d9f223d
consensus-1  |     at AttestationPool.getAggregate (file:///usr/app/packages/beacon-node/src/chain/opPools/attestationPool.ts:136:13)
consensus-1  |     at Object.getAggregatedAttestation (file:///usr/app/packages/beacon-node/src/api/impl/validator/index.ts:1040:47)
consensus-1  |     at processTicksAndRejections (node:internal/process/task_queues:95:5)
consensus-1  |     at Object.handler (file:///usr/app/packages/api/src/utils/server/genericJsonServer.ts:45:23)

Operating system

Linux

Lodestar version or commit hash

v1.5.0 and v1.5.1 tested

[nflaig]

Based on the logs, the request that fails is getAggregatedAttestation, this should only affect aggregates and not attestations.

I've seen this issue before when running Lighthouse VC <> Lodestar BN (#5553) and it happened because the VC requested an aggregate for a slot and data_root but didn't previously submit the attestation for the same via submitPoolAttestations. See #5553 (comment) for a more detailed analysis.

In the end, we fixed the issue on the Lighthouse side (sigp/lighthouse#4774).

I can't think of any change why this would start happening now, maybe there was a change in Vouch behavior? My initial guess would be that Vouch submits the attestation to only one of the connected BNs but fetches aggregates from multiple?

[yorickdowne]

Thanks. Spoke with Jim, I'm getting Vouch trace logs. On the Lodestar side, I tested it on 1.13.0 as well and see it there.

[nflaig]

On the Lodestar side, I tested it on 1.13.0 as well and see it there.

Thanks for confirming this, and helping debugging the issue. We really need to look into running Vouch ourselves or even better, include it in our sim tests.

[yorickdowne]

To make this more fun, it's intermittent. Which suggests a race condition / timing / latency thing?

[nflaig]

Which suggests a race condition / timing / latency thing?

It's unlikely as there is a ~4 second delay between submitting the attestation and requesting an aggregate. The Lodestar debug logs could provide more insights on this as we would see the timings of REST API calls.

[nflaig]

To make this more fun, it's intermittent. Which suggests a race condition / timing / latency thing?

I think have a good guess now on why this happens assuming Vouch implements a similar behavior to what Nimbus does #6634 (comment). (maybe @mcdee could quickly confirm this, haven't gotten around to review Vouch code yet)

So why is it just intermittent? As outlined in the linked comment, while this would never happen if Vouch always submits the attestation to all connected bns, it might also not happen if Lodestar bn received the attestation via gossip published by another node of the same fallback / multi node setup. But this is not guaranteed to be the case and is essentially a race between gossip (+ preparing the aggregate) and the REST API call, hence we are seeing this error intermittently.

The good news is that it's not an actual issue and just noisy logs which will be improved (#6648) in our next release. I would expect that Vouch was able to publish the aggregate attestation as it received the aggregate from another (primary) node.

[mcdee]

Vouch uses a submitter strategy to know where to send data. If this is configured by the user to send to all nodes (the multinode strategy) then all beacon nodes should obtain the attestations (and beacon committee subscriptions).

[nflaig]

Vouch uses a submitter strategy to know where to send data. If this is configured by the user to send to all nodes (the multinode strategy) then all beacon nodes should obtain the attestations (and beacon committee subscriptions).

Thanks for the clarification, this could be easily confirmed by looking at Lodestar bn debug logs as we would see the submitPoolAttestations in the same slot as the getAggregatedAttestation call was done but in that case there would not be an error, the code path for this is really simple and well tested.

[nflaig]

A fix #6668 has been included in our latest release v1.18.0, feel free to reopen if you can still observe this issue.

Vouch uses a submitter strategy to know where to send data. If this is configured by the user to send to all nodes (the multinode strategy) then all beacon nodes should obtain the attestations (and beacon committee subscriptions).

Looks all good from Vouch side, this was a bug in Lodestar