Wrong state in attestation validation

[dapplion]

Describe the bug

packages/beacon-node/src/chain/validation/attestation.ts:81 retrieves the wrong state.

• Must be a state in the same chain as attHeadBlock, but dialed to target.epoch

It causes signature validation to fail if the first slot of a fork is missed. The domain of the state will be wrong, since it will pick the domain of the previous fork, due to updated getDomain logic

[twoeths]

got this error right after CAPELLA_FORK_EPOCH in sepolia

grep -e "Multiple errors on submitPoolAttestations" -rn validator-2023-02-28.log
89874:Feb-28 04:04:52.423[]                error: Error publishing attestations slot=1818624  - Internal Server Error: Multiple errors on submitPoolAttestations
89879:Error:  - Internal Server Error: Multiple errors on submitPoolAttestations
89956:Feb-28 04:05:04.337[]                error: Error publishing attestations slot=1818625  - Internal Server Error: Multiple errors on submitPoolAttestations
89961:Error:  - Internal Server Error: Multiple errors on submitPoolAttestations
90030:Feb-28 04:05:16.368[]                error: Error publishing attestations slot=1818626  - Internal Server Error: Multiple errors on submitPoolAttestations
90035:Error:  - Internal Server Error: Multiple errors on submitPoolAttestations
90106:Feb-28 04:05:28.452[]                error: Error publishing attestations slot=1818627  - Internal Server Error: Multiple errors on submitPoolAttestations
90111:Error:  - Internal Server Error: Multiple errors on submitPoolAttestations
90188:Feb-28 04:05:40.203[]                error: Error publishing attestations slot=1818628  - Internal Server Error: Multiple errors on submitPoolAttestations
90192:Error:  - Internal Server Error: Multiple errors on submitPoolAttestations
90325:Feb-28 04:06:04.226[]                error: Error publishing attestations slot=1818630  - Internal Server Error: Multiple errors on submitPoolAttestations
90329:Error:  - Internal Server Error: Multiple errors on submitPoolAttestations
90395:Feb-28 04:06:16.445[]                error: Error publishing attestations slot=1818631  - Internal Server Error: Multiple errors on submitPoolAttestations
90400:Error:  - Internal Server Error: Multiple errors on submitPoolAttestations
90480:Feb-28 04:06:28.284[]                error: Error publishing attestations slot=1818632  - Internal Server Error: Multiple errors on submitPoolAttestations
90483:Error:  - Internal Server Error: Multiple errors on submitPoolAttestations
90544:Feb-28 04:06:40.343[]                error: Error publishing attestations slot=1818633  - Internal Server Error: Multiple errors on submitPoolAttestations
90552:Error:  - Internal Server Error: Multiple errors on submitPoolAttestations
90732:Feb-28 04:07:16.257[]                error: Error publishing attestations slot=1818636  - Internal Server Error: Multiple errors on submitPoolAttestations
90736:Error:  - Internal Server Error: Multiple errors on submitPoolAttestations
90800:Feb-28 04:07:28.245[]                error: Error publishing attestations slot=1818637  - Internal Server Error: Multiple errors on submitPoolAttestations
90804:Error:  - Internal Server Error: Multiple errors on submitPoolAttestations
90868:Feb-28 04:07:40.416[]                error: Error publishing attestations slot=1818638  - Internal Server Error: Multiple errors on submitPoolAttestations
90876:Error:  - Internal Server Error: Multiple errors on submitPoolAttestations
90968:Feb-28 04:08:04.297[]                error: Error publishing attestations slot=1818640  - Internal Server Error: Multiple errors on submitPoolAttestations
90972:Error:  - Internal Server Error: Multiple errors on submitPoolAttestations
91031:Feb-28 04:08:16.463[]                error: Error publishing attestations slot=1818641  - Internal Server Error: Multiple errors on submitPoolAttestations
91035:Error:  - Internal Server Error: Multiple errors on submitPoolAttestations
91108:Feb-28 04:08:28.274[]                error: Error publishing attestations slot=1818642  - Internal Server Error: Multiple errors on submitPoolAttestations
91112:Error:  - Internal Server Error: Multiple errors on submitPoolAttestations
91241:Feb-28 04:08:52.477[]                error: Error publishing attestations slot=1818644  - Internal Server Error: Multiple errors on submitPoolAttestations
91245:Error:  - Internal Server Error: Multiple errors on submitPoolAttestations
91426:Feb-28 04:09:28.234[]                error: Error publishing attestations slot=1818647  - Internal Server Error: Multiple errors on submitPoolAttestations
91430:Error:  - Internal Server Error: Multiple errors on submitPoolAttestations
91550:Feb-28 04:09:52.268[]                error: Error publishing attestations slot=1818649  - Internal Server Error: Multiple errors on submitPoolAttestations
91555:Error:  - Internal Server Error: Multiple errors on submitPoolAttestations
91611:Feb-28 04:10:04.289[]                error: Error publishing attestations slot=1818650  - Internal Server Error: Multiple errors on submitPoolAttestations
91614:Error:  - Internal Server Error: Multiple errors on submitPoolAttestations
91681:Feb-28 04:10:16.535[]                error: Error publishing attestations slot=1818651  - Internal Server Error: Multiple errors on submitPoolAttestations
91687:Error:  - Internal Server Error: Multiple errors on submitPoolAttestations
91767:Feb-28 04:10:28.366[]                error: Error publishing attestations slot=1818652  - Internal Server Error: Multiple errors on submitPoolAttestations
91774:Error:  - Internal Server Error: Multiple errors on submitPoolAttestations
91857:Feb-28 04:10:40.264[]                error: Error publishing attestations slot=1818653  - Internal Server Error: Multiple errors on submitPoolAttestations
91865:Error:  - Internal Server Error: Multiple errors on submitPoolAttestations
91934:Feb-28 04:10:52.253[]                error: Error publishing attestations slot=1818654  - Internal Server Error: Multiple errors on submitPoolAttestations
91938:Error:  - Internal Server Error: Multiple errors on submitPoolAttestations
91997:Feb-28 04:11:04.308[]                error: Error publishing attestations slot=1818655  - Internal Server Error: Multiple errors on submitPoolAttestations
92002:Error:  - Internal Server Error: Multiple errors on submitPoolAttestations

[twoeths]

it has happened until node is synced. Stack trace of the above error:

Feb-28 04:11:04.308[]                ^[[31merror^[[39m: Error publishing attestations slot=1818655  - Internal Server Error: Multiple errors on submitPoolAttestations
ATTESTATION_ERROR_INVALID_SIGNATURE
ATTESTATION_ERROR_INVALID_SIGNATURE
ATTESTATION_ERROR_INVALID_SIGNATURE
ATTESTATION_ERROR_INVALID_SIGNATURE
Error:  - Internal Server Error: Multiple errors on submitPoolAttestations
ATTESTATION_ERROR_INVALID_SIGNATURE
ATTESTATION_ERROR_INVALID_SIGNATURE
ATTESTATION_ERROR_INVALID_SIGNATURE
ATTESTATION_ERROR_INVALID_SIGNATURE
    at Function.assert (file:///usr/app/packages/api/src/utils/client/httpClient.ts:44:13)
    at AttestationService.signAndPublishAttestations (file:///usr/app/packages/validator/src/services/attestation.ts:156:16)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at AttestationService.runAttestationTasks (file:///usr/app/packages/validator/src/services/attestation.ts:74:5)
    at Clock.runAtMostEvery (file:///usr/app/packages/validator/src/util/clock.ts:96:7)