Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Error updating eth1 chain cache message=Invalid length index (kintsugi-lodestar-geth/nethermind) #3597

Closed
SeaMonkey82 opened this issue Jan 9, 2022 · 15 comments
Closed

Error updating eth1 chain cache message=Invalid length index (kintsugi-lodestar-geth/nethermind) #3597

SeaMonkey82 opened this issue Jan 9, 2022 · 15 comments
Assignees
@g11tech
Labels
spec-bellatrix 🐼 Issues targeting the merge spec version.

Comments

@SeaMonkey82
Copy link

SeaMonkey82 commented Jan 9, 2022 •
edited

Describe the bug

Flood of this error
Error updating eth1 chain cache message=Invalid length index

Expected behavior

non-error output

Steps to Reproduce

Sync kintsugi-lodestar-geth or kintsugi-lodestar-nethermind (maybe all EL clients?)

Screenshots Output

Error: Invalid length index
    at CompositeListType.tree_setProperty (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/node_modules/@chainsafe/ssz/src/types/composite/list.ts:494:13)
    at CompositeListTreeValue.setProperty (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/node_modules/@chainsafe/ssz/src/backings/tree/treeValue.ts:294:22)
    at Object.set (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/node_modules/@chainsafe/ssz/src/backings/tree/treeValue.ts:92:19)
    at DepositDataRootRepository.batchPut (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/lodestar/src/db/repositories/depositDataRoot.ts:34:27)
    at DepositDataRootRepository.batchPutValues (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/lodestar/src/db/repositories/depositDataRoot.ts:43:5)
    at Eth1DepositsCache.add (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/lodestar/src/eth1/eth1DepositsCache.ts:59:5)
    at Eth1DepositDataTracker.updateDepositCache (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/lodestar/src/eth1/eth1DepositDataTracker.ts:163:5)
    at Eth1DepositDataTracker.update (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/lodestar/src/eth1/eth1DepositDataTracker.ts:144:33)
    at Eth1DepositDataTracker.runAutoUpdate (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/lodestar/src/eth1/eth1DepositDataTracker.ts:118:29)

Desktop (please complete the following information):

  • OS: Linux Mint
  • Version: 20.3
  • Branch: master
  • Commit hash: 3e8ed5c06c6ff69f3310366c974f68e33cba856e
@philknows
Copy link
Member

Interesting... haven't seen this one before @g11tech.

@g11tech
Copy link
Contributor

g11tech commented Jan 14, 2022 •
edited

@SeaMonkey82 is this coming now/reproducible? couldn't find this in the log of the recently updated kintsugi nodes

@philknows seems like deposit data is not correct, not a kintsugi issue though, but could be because of bad blocks that are being pushed into network as part of robustness/stress testing.

@SeaMonkey82
Copy link
Author

SeaMonkey82 commented Jan 14, 2022 •
edited

edit Different error now.

➜  lodestar git:(master) ./lodestar --version
🌟 Lodestar: TypeScript Implementation of the Ethereum 2.0 Beacon Chain.
  * Version: v0.32.0/master/+85/b4facd38 (git)

Jan-14 07:16:08.737 [NETWORK]         error: Error adding aggregated attestation to forkchoice slot=208555 code=FORKCHOICE_ERROR_INVALID_ATTESTATION, err={"code":"FUTURE_SLOT","attestationSlot":208555,"latestPermissibleSlot":208554}
Error: FORKCHOICE_ERROR_INVALID_ATTESTATION
    at ForkChoice.validateAttestationData (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/fork-choice/src/forkChoice/forkChoice.ts:808:13)
    at ForkChoice.validateOnAttestation (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/fork-choice/src/forkChoice/forkChoice.ts:763:10)
    at ForkChoice.onAttestation (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/fork-choice/src/forkChoice/forkChoice.ts:421:10)
    at beacon_aggregate_and_proof (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/lodestar/src/network/gossip/handlers/index.ts:173:30)
    at Object.gossipValidatorFn [as beacon_aggregate_and_proof] (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/lodestar/src/network/gossip/validation/index.ts:110:7)
    at Eth2Gossipsub.validate (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/lodestar/src/network/gossip/gossipsub.ts:282:7)
    at Eth2Gossipsub._processRpcMessage (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/node_modules/libp2p-interfaces/src/pubsub/index.js:449:7)

@g11tech
Copy link
Contributor

g11tech commented Jan 14, 2022

this error looks legit, as the attestation arrive for a future slot it seems as we currently reject future slot attestation/arriving early than expected.

@SeaMonkey82
Copy link
Author 

Jan-14 14:24:03.405 [ETH1]            error: Error updating eth1 chain cache message=Invalid length index
Error: Invalid length index
    at CompositeListType.tree_setProperty (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/node_modules/@chainsafe/ssz/src/types/composite/list.ts:494:13)
    at CompositeListTreeValue.setProperty (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/node_modules/@chainsafe/ssz/src/backings/tree/treeValue.ts:294:22)
    at Object.set (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/node_modules/@chainsafe/ssz/src/backings/tree/treeValue.ts:92:19)
    at DepositDataRootRepository.batchPut (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/lodestar/src/db/repositories/depositDataRoot.ts:34:27)
    at DepositDataRootRepository.batchPutValues (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/lodestar/src/db/repositories/depositDataRoot.ts:43:5)
    at Eth1DepositsCache.add (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/lodestar/src/eth1/eth1DepositsCache.ts:59:5)
    at Eth1DepositDataTracker.updateDepositCache (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/lodestar/src/eth1/eth1DepositDataTracker.ts:163:5)
    at Eth1DepositDataTracker.update (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/lodestar/src/eth1/eth1DepositDataTracker.ts:144:33)
    at Eth1DepositDataTracker.runAutoUpdate (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/lodestar/src/eth1/eth1DepositDataTracker.ts:118:29)

@g11tech
Copy link
Contributor

g11tech commented Jan 15, 2022 

Jan-14 14:24:03.405 [ETH1]            error: Error updating eth1 chain cache message=Invalid length index
Error: Invalid length index
    at CompositeListType.tree_setProperty (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/node_modules/@chainsafe/ssz/src/types/composite/list.ts:494:13)
    at CompositeListTreeValue.setProperty (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/node_modules/@chainsafe/ssz/src/backings/tree/treeValue.ts:294:22)
    at Object.set (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/node_modules/@chainsafe/ssz/src/backings/tree/treeValue.ts:92:19)
    at DepositDataRootRepository.batchPut (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/lodestar/src/db/repositories/depositDataRoot.ts:34:27)
    at DepositDataRootRepository.batchPutValues (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/lodestar/src/db/repositories/depositDataRoot.ts:43:5)
    at Eth1DepositsCache.add (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/lodestar/src/eth1/eth1DepositsCache.ts:59:5)
    at Eth1DepositDataTracker.updateDepositCache (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/lodestar/src/eth1/eth1DepositDataTracker.ts:163:5)
    at Eth1DepositDataTracker.update (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/lodestar/src/eth1/eth1DepositDataTracker.ts:144:33)
    at Eth1DepositDataTracker.runAutoUpdate (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/lodestar/src/eth1/eth1DepositDataTracker.ts:118:29)

Hi @SeaMonkey82 Since am not able to see this on our nodes. Could you help me with these? 🙂
which commit hash for lodestar, geth/nethermind (or better yet docker images) ?
Is it a random occurrence or do you see this consistently ?
could you include a bit more of log above and below this error?

@SeaMonkey82
Copy link
Author

SeaMonkey82 commented Jan 15, 2022 •
edited 

➜  lodestar git:(master) ./lodestar --version
🌟 Lodestar: TypeScript Implementation of the Ethereum 2.0 Beacon Chain.
  * Version: v0.32.0/master/+85/b4facd38 (git)

➜  lodestar git:(master) geth-kintsugi version
Geth
Version: 1.10.16-unstable
Git Commit: 63a89820ca9721ef2256859391db5a5920faf16c
Git Commit Date: 20220110
Architecture: amd64
Go Version: go1.17.5
Operating System: linux

➜  Nethermind.Runner git:(b565ffba0) ✗ dotnet run -c Release -- --version                                                                                               
2022-01-15 08-16-34.7059|Nethermind starting initialization.
2022-01-15 08-16-34.7367|Loading embedded plugins
2022-01-15 08-16-34.7368|  Found plugin type Nethermind.Consensus.AuRa.AuRaPlugin
2022-01-15 08-16-34.7368|  Found plugin type Nethermind.Consensus.Clique.CliquePlugin
2022-01-15 08-16-34.7368|  Found plugin type Nethermind.Consensus.Ethash.EthashPlugin
2022-01-15 08-16-34.7368|  Found plugin type Nethermind.Consensus.Ethash.NethDevPlugin
Resolved executing directory as /home/seamonkey/Downloads/merge-testnets/kintsugi/nethermind/src/Nethermind/Nethermind.Runner/bin/Release/net6.0.
2022-01-15 08-16-34.7437|Loading 7 assemblies from /home/seamonkey/Downloads/merge-testnets/kintsugi/nethermind/src/Nethermind/Nethermind.Runner/bin/Release/net6.0/plugins
2022-01-15 08-16-34.7438|Loading assembly Nethermind.HealthChecks
2022-01-15 08-16-34.7450|  Found plugin type Nethermind.HealthChecks
2022-01-15 08-16-34.7450|Loading assembly Nethermind.Merge.Plugin
2022-01-15 08-16-34.7465|  Found plugin type Nethermind.Merge.Plugin
2022-01-15 08-16-34.7465|Loading assembly Nethermind.Consensus.AuRa
2022-01-15 08-16-34.7488|Loading assembly Nethermind.Api
2022-01-15 08-16-34.7491|Loading assembly Nethermind.Mev
2022-01-15 08-16-34.7504|  Found plugin type Nethermind.Mev
2022-01-15 08-16-34.7504|Loading assembly Nethermind.Consensus.Ethash
2022-01-15 08-16-34.7505|Loading assembly Nethermind.Consensus.Clique

Nethermind/v1.11.7-409-b565ffba0-20220115/X64-Linux/6.0.1

Occurs consistently. Here is the log from lodestar-geth. lodestar-nethermind looks similar.

Jan-15 08:13:44.366 [NETWORK]         error: Error adding aggregated attestation to forkchoice slot=216043 code=FORKCHOICE_ERROR_INVALID_ATTESTATION, err={"code":"FUTURE_SLOT","attestationSlot":216043,"latestPermissibleSlot":216042}
Error: FORKCHOICE_ERROR_INVALID_ATTESTATION
    at ForkChoice.validateAttestationData (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/fork-choice/src/forkChoice/forkChoice.ts:808:13)
    at ForkChoice.validateOnAttestation (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/fork-choice/src/forkChoice/forkChoice.ts:763:10)
    at ForkChoice.onAttestation (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/fork-choice/src/forkChoice/forkChoice.ts:421:10)
    at beacon_aggregate_and_proof (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/lodestar/src/network/gossip/handlers/index.ts:173:30)
    at Object.gossipValidatorFn [as beacon_aggregate_and_proof] (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/lodestar/src/network/gossip/validation/index.ts:110:7)
    at Eth2Gossipsub.validate (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/lodestar/src/network/gossip/gossipsub.ts:282:7)
    at Eth2Gossipsub._processRpcMessage (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/node_modules/libp2p-interfaces/src/pubsub/index.js:449:7)
Jan-15 08:13:44.367 [NETWORK]         error: Error adding aggregated attestation to forkchoice slot=216043 code=FORKCHOICE_ERROR_INVALID_ATTESTATION, err={"code":"FUTURE_SLOT","attestationSlot":216043,"latestPermissibleSlot":216042}
Error: FORKCHOICE_ERROR_INVALID_ATTESTATION
    at ForkChoice.validateAttestationData (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/fork-choice/src/forkChoice/forkChoice.ts:808:13)
    at ForkChoice.validateOnAttestation (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/fork-choice/src/forkChoice/forkChoice.ts:763:10)
    at ForkChoice.onAttestation (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/fork-choice/src/forkChoice/forkChoice.ts:421:10)
    at beacon_aggregate_and_proof (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/lodestar/src/network/gossip/handlers/index.ts:173:30)
    at Object.gossipValidatorFn [as beacon_aggregate_and_proof] (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/lodestar/src/network/gossip/validation/index.ts:110:7)
    at Eth2Gossipsub.validate (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/lodestar/src/network/gossip/gossipsub.ts:282:7)
    at Eth2Gossipsub._processRpcMessage (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/node_modules/libp2p-interfaces/src/pubsub/index.js:449:7)
Jan-15 08:13:44.367 [NETWORK]         error: Error adding aggregated attestation to forkchoice slot=216043 code=FORKCHOICE_ERROR_INVALID_ATTESTATION, err={"code":"FUTURE_SLOT","attestationSlot":216043,"latestPermissibleSlot":216042}
Error: FORKCHOICE_ERROR_INVALID_ATTESTATION
    at ForkChoice.validateAttestationData (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/fork-choice/src/forkChoice/forkChoice.ts:808:13)
    at ForkChoice.validateOnAttestation (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/fork-choice/src/forkChoice/forkChoice.ts:763:10)
    at ForkChoice.onAttestation (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/fork-choice/src/forkChoice/forkChoice.ts:421:10)
    at beacon_aggregate_and_proof (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/lodestar/src/network/gossip/handlers/index.ts:173:30)
    at Object.gossipValidatorFn [as beacon_aggregate_and_proof] (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/lodestar/src/network/gossip/validation/index.ts:110:7)
    at Eth2Gossipsub.validate (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/lodestar/src/network/gossip/gossipsub.ts:282:7)
    at Eth2Gossipsub._processRpcMessage (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/node_modules/libp2p-interfaces/src/pubsub/index.js:449:7)
Jan-15 08:13:44.519 [ETH1]            error: Error updating eth1 chain cache message=Invalid length index
Error: Invalid length index
    at CompositeListType.tree_setProperty (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/node_modules/@chainsafe/ssz/src/types/composite/list.ts:494:13)
    at CompositeListTreeValue.setProperty (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/node_modules/@chainsafe/ssz/src/backings/tree/treeValue.ts:294:22)
    at Object.set (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/node_modules/@chainsafe/ssz/src/backings/tree/treeValue.ts:92:19)
    at DepositDataRootRepository.batchPut (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/lodestar/src/db/repositories/depositDataRoot.ts:34:27)
    at DepositDataRootRepository.batchPutValues (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/lodestar/src/db/repositories/depositDataRoot.ts:43:5)
    at Eth1DepositsCache.add (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/lodestar/src/eth1/eth1DepositsCache.ts:59:5)
    at Eth1DepositDataTracker.updateDepositCache (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/lodestar/src/eth1/eth1DepositDataTracker.ts:163:5)
    at Eth1DepositDataTracker.update (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/lodestar/src/eth1/eth1DepositDataTracker.ts:144:33)
    at Eth1DepositDataTracker.runAutoUpdate (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/lodestar/src/eth1/eth1DepositDataTracker.ts:118:29)
Jan-15 08:13:44.659 [NETWORK]         error: Error adding aggregated attestation to forkchoice slot=216043 code=FORKCHOICE_ERROR_INVALID_ATTESTATION, err={"code":"FUTURE_SLOT","attestationSlot":216043,"latestPermissibleSlot":216042}
Error: FORKCHOICE_ERROR_INVALID_ATTESTATION
    at ForkChoice.validateAttestationData (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/fork-choice/src/forkChoice/forkChoice.ts:808:13)
    at ForkChoice.validateOnAttestation (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/fork-choice/src/forkChoice/forkChoice.ts:763:10)
    at ForkChoice.onAttestation (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/fork-choice/src/forkChoice/forkChoice.ts:421:10)
    at beacon_aggregate_and_proof (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/lodestar/src/network/gossip/handlers/index.ts:173:30)
    at Object.gossipValidatorFn [as beacon_aggregate_and_proof] (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/lodestar/src/network/gossip/validation/index.ts:110:7)
    at Eth2Gossipsub.validate (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/lodestar/src/network/gossip/gossipsub.ts:282:7)
    at Eth2Gossipsub._processRpcMessage (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/node_modules/libp2p-interfaces/src/pubsub/index.js:449:7)
Jan-15 08:13:44.659 [NETWORK]         error: Error adding aggregated attestation to forkchoice slot=216043 code=FORKCHOICE_ERROR_INVALID_ATTESTATION, err={"code":"FUTURE_SLOT","attestationSlot":216043,"latestPermissibleSlot":216042}
Error: FORKCHOICE_ERROR_INVALID_ATTESTATION
    at ForkChoice.validateAttestationData (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/fork-choice/src/forkChoice/forkChoice.ts:808:13)
    at ForkChoice.validateOnAttestation (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/fork-choice/src/forkChoice/forkChoice.ts:763:10)
    at ForkChoice.onAttestation (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/fork-choice/src/forkChoice/forkChoice.ts:421:10)
    at beacon_aggregate_and_proof (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/lodestar/src/network/gossip/handlers/index.ts:173:30)
    at Object.gossipValidatorFn [as beacon_aggregate_and_proof] (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/lodestar/src/network/gossip/validation/index.ts:110:7)
    at Eth2Gossipsub.validate (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/lodestar/src/network/gossip/gossipsub.ts:282:7)
    at Eth2Gossipsub._processRpcMessage (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/node_modules/libp2p-interfaces/src/pubsub/index.js:449:7)
Jan-15 08:13:44.660 [NETWORK]         error: Error adding aggregated attestation to forkchoice slot=216043 code=FORKCHOICE_ERROR_INVALID_ATTESTATION, err={"code":"FUTURE_SLOT","attestationSlot":216043,"latestPermissibleSlot":216042}
Error: FORKCHOICE_ERROR_INVALID_ATTESTATION
    at ForkChoice.validateAttestationData (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/fork-choice/src/forkChoice/forkChoice.ts:808:13)
    at ForkChoice.validateOnAttestation (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/fork-choice/src/forkChoice/forkChoice.ts:763:10)
    at ForkChoice.onAttestation (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/fork-choice/src/forkChoice/forkChoice.ts:421:10)
    at beacon_aggregate_and_proof (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/lodestar/src/network/gossip/handlers/index.ts:173:30)
    at Object.gossipValidatorFn [as beacon_aggregate_and_proof] (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/lodestar/src/network/gossip/validation/index.ts:110:7)
    at Eth2Gossipsub.validate (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/packages/lodestar/src/network/gossip/gossipsub.ts:282:7)
    at Eth2Gossipsub._processRpcMessage (/home/seamonkey/Downloads/merge-testnets/kintsugi/lodestar/node_modules/libp2p-interfaces/src/pubsub/index.js:449:7)
Jan-15 08:13:44.797 [NETWORK]         error: Error adding aggregated attestation to forkchoice slot=216043 code=FORKCHOICE_ERROR_INVALID_ATTESTATION, err={"code":"FUTURE_SLOT","attestationSlot":216043,"latestPermissibleSlot":216042}

@g11tech
Copy link
Contributor

g11tech commented Jan 15, 2022

@SeaMonkey82 thanks! 🙂 I will try to repro at my end. I am not that worried about invalid attestation as an attestation could potentially arrive earlier than block causing this error (but I will still try validating if this is the case). In future lodestar could queue this attestation up to process later.

However I am more curious about the reasons behind Invalid length index which seems to be occurring while saving deposit. 🤔

@SeaMonkey82
Copy link
Author

Update: This error was caused by including --network mainnet in my validator launch command. I tested by removing it and all three lodestar instances (with geth, nethermind, and besu) stopped seeing the error. I don't have a good reason for why that flag was there, but curiously enough, the only thing it broke was block proposal. Attestations worked fine.

Leaving this issue open in case of any further discussion or action. It seems to me that --network mainnet should conflict with --paramsFile="../config.yaml" (kintsugi in this case) and automatically exit with a message describing the conflict.

@g11tech
Copy link
Contributor

g11tech commented Jan 21, 2022

Update: This error was caused by including --network mainnet in my validator launch command. I tested by removing it and all three lodestar instances (with geth, nethermind, and besu) stopped seeing the error. I don't have a good reason for why that flag was there, but curiously enough, the only thing it broke was block proposal. Attestations worked fine.

Leaving this issue open in case of any further discussion or action. It seems to me that --network mainnet should conflict with --paramsFile="../config.yaml" (kintsugi in this case) and automatically exit with a message describing the conflict.

Thanks @SeaMonkey82 . Yes, the validator does validate the config it fetches from beacon against its own config.yaml. will check it out why it hasn't happened here.

@SeaMonkey82
Copy link
Author

SeaMonkey82 commented Jan 21, 2022 •
edited

I guess the fact that these messages stopped after restarting my validators without --network mainnet was a coincidence. All three beacon nodes are doing it again.

@blampe
Copy link

blampe commented Jan 24, 2022

I also saw this on my first run of chainsafe/lodestar:v0.33.0 using Prater. Command:

./node_modules/.bin/lodestar beacon --network=prater --metrics.enabled=true --metrics.serverPort=8008 --rootDir=/data --network.maxPeers=30 --eth1.providerUrls http://nethermind:8545 http://erigon:8545

This was reproducible after wiping the data directory and re-running.

I suspect there's a bad interaction between Lodestar/Nethermind, because after switching the order of eth1 providers (presumably to give Erigon priority over Nethermind) the problem went away. (Although I then encounter #3550 and something to do with BLOCK_ERROR_PARENT_UNKNOWN.)

FWIW, I have Nethermind using their recommended configuration for validators: --Pruning.Enabled=true --Sync.DownloadBodiesInFastSync=true --Sync.DownloadReceiptsInFastSync=true --Sync.AncientBodiesBarrier=11052984 --Sync.AncientReceiptsBarrier=11052984.

@philknows philknows added the spec-bellatrix 🐼 Issues targeting the merge spec version. label Jan 24, 2022
@g11tech
Copy link
Contributor

g11tech commented Jan 24, 2022

thank you for the update! so it seems that it has nothing to do with the kintsugi/bellatrix but deposit tracker interaction with the EL. let me try to replicate using the nethermind master

@g11tech
Copy link
Contributor

g11tech commented Jan 24, 2022 •
edited

I also saw this on my first run of chainsafe/lodestar:v0.33.0 using Prater. Command:

./node_modules/.bin/lodestar beacon --network=prater --metrics.enabled=true --metrics.serverPort=8008 --rootDir=/data --network.maxPeers=30 --eth1.providerUrls http://nethermind:8545 http://erigon:8545

This was reproducible after wiping the data directory and re-running.

I suspect there's a bad interaction between Lodestar/Nethermind, because after switching the order of eth1 providers (presumably to give Erigon priority over Nethermind) the problem went away. (Although I then encounter #3550 and something to do with BLOCK_ERROR_PARENT_UNKNOWN.)

FWIW, I have Nethermind using their recommended configuration for validators: --Pruning.Enabled=true --Sync.DownloadBodiesInFastSync=true --Sync.DownloadReceiptsInFastSync=true --Sync.AncientBodiesBarrier=11052984 --Sync.AncientReceiptsBarrier=11052984.

Could you provide nethermind branch + commit /docker image details along with the (exact if possible) command that you used to spin up nethermind?

UPDATE: I was able to run lodestar on prater with eth1 as nethermind without any issue:

an-24 18:10:42.002 []                 info: Synced - slot: 2211653 - head: 2211653 0xa8a9…6162 - finalized: 0xcf0f…1522:69112 - peers: 9
Jan-24 18:10:54.001 []                 info: Synced - slot: 2211654 - head: 2211654 0xd4ff…003b - finalized: 0xcf0f…1522:69112 - peers: 10
Jan-24 18:11:06.001 []                 info: Synced - slot: 2211655 - head: 2211655 0x7f1f…6a3c - finalized: 0xcf0f…1522:69112 - peers: 10
Jan-24 18:11:17.999 []                 info: Synced - slot: 2211656 - head: 2211656 0x645b…7011 - finalized: 0xcf0f…1522:69112 - peers: 13
Jan-24 18:11:18.001 []                 info: Synced - slot: 2211656 - head: 2211656 0x645b…7011 - finalized: 0xcf0f…1522:69112 - peers: 13
Jan-24 18:11:30.000 []                 info: Synced - slot: 2211657 - head: 2211657 0x322a…3dc4 - finalized: 0xcf0f…1522:69112 - peers: 18
Jan-24 18:11:42.001 []                 info: Synced - slot: 2211658 - head: 2211658 0x8d9b…eb85 - finalized: 0xcf0f…1522:69112 - peers: 18
Jan-24 18:11:54.000 []                 info: Synced - slot: 2211659 - head: 2211659 0xe843…5a02 - finalized: 0xcf0f…1522:69112 - peers: 19
Jan-24 18:12:0

nethermind args:
--config goerli --datadir /data/nethermind --JsonRpc.Enabled=true --JsonRpc.EnabledModules=net,eth,consensus,engine,subscribe,web3 --JsonRpc.Port=8545 --JsonRpc.WebSocketsPort=8546 --JsonRpc.Host=0.0.0.0

@g11tech
Copy link
Contributor

g11tech commented Jan 27, 2022 •
edited

@SeaMonkey82 @blampe

Et voila! This annoying error is now RESOLVED!!! 🥳
@SeaMonkey82 you might not be setting --eth1.depositContractDeployBlock argument which for kintsugi should be set to --eth1.depositContractDeployBlock 0. This causes the lodestar deposit tracker to fetch skipped data, from which deposit root can't be derived and hence the error!

And you @blampe seems to have mixed up the networks on CL and EL side, so this error could again comeup because of wrong/unexpected feed of the deposit roots resulting again to the scenario described above!

Anyway, now one doesn't need to worry about configuration, kintsugi configuration can now automatically be loaded with --network kintsugi (you still have to specify eth1... and execution... params to interface the EL)

cc: @philknows "repro-ing" this error was harder than solving it 🤣

special thanks to Nethermind's @MarekM25 and his teammate @AntiD2ta for helping repro this error!

@g11tech g11tech closed this as completed Jan 27, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@g11tech g11tech

Labels
spec-bellatrix 🐼 Issues targeting the merge spec version.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@blampe @SeaMonkey82 @philknows @g11tech