Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Corrupted deposits history detected #4227

Closed
jfblaine opened this issue Oct 7, 2022 · 3 comments
Closed

Corrupted deposits history detected #4227

jfblaine opened this issue Oct 7, 2022 · 3 comments

Comments

@jfblaine
Copy link

jfblaine commented Oct 7, 2022 •
edited

I noticed the Eth1 chain length in Grafana is no longer a consistent sawtooth pattern and now looks like this:

image

It appears the change in pattern coincides with the first appearance of the "Corrupted deposits history detected" message. This pattern repeats however I am not currently missing any attestations.

Oct 07 04:34:51 hostname nimbus_beacon_node[748]: NOT 2022-10-07 04:34:51.027-04:00 Attestation sent attestation="(aggregation_bits: 0b000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000000000000, data: (slot: 4858972, index: 17, beacon_block_root: "071bd883", source: "151841:e75a516c", target: "151842:ccb3be9a"), signature: "82980f49")" delay=25ms795us228ns subnet_id=17
Oct 07 04:34:55 hostname nimbus_beacon_node[748]: INF 2022-10-07 04:34:55.049-04:00 Slot end topics="beacnde" slot=4858972 nextActionWait=4m39s950ms425us323ns nextAttestationSlot=4858996 nextProposalSlot=-1 syncCommitteeDuties=none head=071bd883:4858972
Oct 07 04:34:59 hostname nimbus_beacon_node[748]: INF 2022-10-07 04:34:59.000-04:00 Slot start topics="beacnde" slot=4858973 epoch=151842 sync=synced peers=157 head=071bd883:4858972 finalized=151840:6ee4a35f delay=218us358ns
Oct 07 04:35:07 hostname nimbus_beacon_node[748]: INF 2022-10-07 04:35:07.044-04:00 Slot end topics="beacnde" slot=4858973 nextActionWait=4m27s955ms883us524ns nextAttestationSlot=4858996 nextProposalSlot=-1 syncCommitteeDuties=none head=dba2fea0:4858973
Oct 07 04:35:11 hostname nimbus_beacon_node[748]: INF 2022-10-07 04:35:11.000-04:00 Slot start topics="beacnde" slot=4858974 epoch=151842 sync=synced peers=157 head=dba2fea0:4858973 finalized=151840:6ee4a35f delay=190us819ns
Oct 07 04:35:19 hostname nimbus_beacon_node[748]: INF 2022-10-07 04:35:19.050-04:00 Slot end topics="beacnde" slot=4858974 nextActionWait=4m15s949ms746us337ns nextAttestationSlot=4858996 nextProposalSlot=-1 syncCommitteeDuties=none head=8d2d761e:4858974
Oct 07 04:35:23 hostname nimbus_beacon_node[748]: INF 2022-10-07 04:35:23.000-04:00 Slot start topics="beacnde" slot=4858975 epoch=151842 sync=synced peers=157 head=8d2d761e:4858974 finalized=151840:6ee4a35f delay=575us281ns
Oct 07 04:35:31 hostname nimbus_beacon_node[748]: INF 2022-10-07 04:35:31.024-04:00 Slot end topics="beacnde" slot=4858975 nextActionWait=4m3s975ms73us40ns nextAttestationSlot=4858996 nextProposalSlot=-1 syncCommitteeDuties=none head=9a92ee43:4858975
Oct 07 04:35:35 hostname nimbus_beacon_node[748]: INF 2022-10-07 04:35:35.000-04:00 Slot start topics="beacnde" slot=4858976 epoch=151843 sync=synced peers=157 head=9a92ee43:4858975 finalized=151840:6ee4a35f delay=638us430ns
Oct 07 04:35:38 hostname nimbus_beacon_node[748]: ERR 2022-10-07 04:35:38.064-04:00 Corrupted deposits history detected topics="eth1" ourDepositsCount=450876 taretDepositsCount=450876 ourDepositsRoot=cde455fc targetDepositsRoot=ace544b6
Oct 07 04:35:43 hostname nimbus_beacon_node[748]: INF 2022-10-07 04:35:43.488-04:00 State replayed topics="chaindag" blocks=0 slots=32 current=9a92ee43:4858975@4858976 ancestor=a8ba962e:4858976 target=a8ba962e:4858976@4859008 ancestorStateRoot=6c4a3d06 targetStateRoot=295e75d2 found=true assignDur=6ms966us571ns replayDur=409ms70us11ns
Oct 07 04:35:43 hostname nimbus_beacon_node[748]: INF 2022-10-07 04:35:43.503-04:00 Slot end topics="beacnde" slot=4858976 nextActionWait=3m51s496ms970us469ns nextAttestationSlot=4858996 nextProposalSlot=-1 syncCommitteeDuties=none head=a8ba962e:4858976
Oct 07 04:35:46 hostname nimbus_beacon_node[748]: WRN 2022-10-07 04:35:46.255-04:00 Eth1 chain monitoring failure, restarting topics="eth1" err="Eth1 chain contradicts Eth2 consensus"
Oct 07 04:35:47 hostname nimbus_beacon_node[748]: INF 2022-10-07 04:35:47.000-04:00 Slot start topics="beacnde" slot=4858977 epoch=151843 sync=synced peers=157 head=a8ba962e:4858976 finalized=151841:e75a516c delay=861us257ns
Oct 07 04:35:51 hostname nimbus_beacon_node[748]: INF 2022-10-07 04:35:51.257-04:00 Established connection to execution layer topics="eth1" url=http://127.0.0.1:8551
Oct 07 04:35:51 hostname nimbus_beacon_node[748]: INF 2022-10-07 04:35:51.259-04:00 Starting Eth1 deposit contract monitoring topics="eth1" contract=0x00000000219ab540356cbb839cbe05303d7705fa
Oct 07 04:35:55 hostname nimbus_beacon_node[748]: INF 2022-10-07 04:35:55.044-04:00 Slot end topics="beacnde" slot=4858977 nextActionWait=3m39s955ms578us748ns nextAttestationSlot=4858996 nextProposalSlot=-1 syncCommitteeDuties=none head=b006426d:4858977
Oct 07 04:35:59 hostname nimbus_beacon_node[748]: INF 2022-10-07 04:35:59.000-04:00 Slot start topics="beacnde" slot=4858978 epoch=151843 sync=synced peers=157 head=b006426d:4858977 finalized=151841:e75a516c delay=453us26ns
Oct 07 04:36:07 hostname nimbus_beacon_node[748]: INF 2022-10-07 04:36:07.040-04:00 Slot end topics="beacnde" slot=4858978 nextActionWait=3m27s959ms853us129ns nextAttestationSlot=4858996 nextProposalSlot=-1 syncCommitteeDuties=none head=52488d8f:4858978
Oct 07 04:36:11 hostname nimbus_beacon_node[748]: INF 2022-10-07 04:36:11.000-04:00 Slot start topics="beacnde" slot=4858979 epoch=151843 sync=synced peers=157 head=52488d8f:4858978 finalized=151841:e75a516c delay=188us631ns
Oct 07 04:36:19 hostname nimbus_beacon_node[748]: INF 2022-10-07 04:36:19.050-04:00 Slot end topics="beacnde" slot=4858979 nextActionWait=3m15s949ms139us683ns nextAttestationSlot=4858996 nextProposalSlot=-1 syncCommitteeDuties=none head=e88d3e54:4858979
Oct 07 04:36:23 hostname nimbus_beacon_node[748]: INF 2022-10-07 04:36:23.000-04:00 Slot start topics="beacnde" slot=4858980 epoch=151843 sync=synced peers=157 head=e88d3e54:4858979 finalized=151841:e75a516c delay=940us965ns
Oct 07 04:36:31 hostname nimbus_beacon_node[748]: INF 2022-10-07 04:36:31.051-04:00 Slot end topics="beacnde" slot=4858980 nextActionWait=3m3s948ms425us665ns nextAttestationSlot=4858996 nextProposalSlot=-1 syncCommitteeDuties=none head=e4bd98ea:4858980
Oct 07 04:36:35 hostname nimbus_beacon_node[748]: INF 2022-10-07 04:36:35.000-04:00 Slot start topics="beacnde" slot=4858981 epoch=151843 sync=synced peers=157 head=e4bd98ea:4858980 finalized=151841:e75a516c delay=721us292ns
Oct 07 04:36:42 hostname nimbus_beacon_node[748]: INF 2022-10-07 04:36:42.835-04:00 Eth1 sync progress topics="eth1" blockNumber=15693002 depositsProcessed=451016
Oct 07 04:36:43 hostname nimbus_beacon_node[748]: INF 2022-10-07 04:36:43.020-04:00 Slot end topics="beacnde" slot=4858981 nextActionWait=2m51s979ms931us41ns nextAttestationSlot=4858996 nextProposalSlot=-1 syncCommitteeDuties=none head=3dd32c1c:4858981
Oct 07 04:36:47 hostname nimbus_beacon_node[748]: INF 2022-10-07 04:36:47.001-04:00 Slot start topics="beacnde" slot=4858982 epoch=151843 sync=synced peers=157 head=3dd32c1c:4858981 finalized=151841:e75a516c delay=994us827ns
Oct 07 04:36:55 hostname nimbus_beacon_node[748]: INF 2022-10-07 04:36:55.052-04:00 Slot end topics="beacnde" slot=4858982 nextActionWait=2m39s947ms781us835ns nextAttestationSlot=4858996 nextProposalSlot=-1 syncCommitteeDuties=none head=b9facffe:4858982
Oct 07 04:36:59 hostname nimbus_beacon_node[748]: INF 2022-10-07 04:36:59.000-04:00 Slot start topics="beacnde" slot=4858983 epoch=151843 sync=synced peers=157 head=b9facffe:4858982 finalized=151841:e75a516c delay=329us252ns
Oct 07 04:37:07 hostname nimbus_beacon_node[748]: INF 2022-10-07 04:37:07.047-04:00 Slot end topics="beacnde" slot=4858983 nextActionWait=2m27s952ms861us551ns nextAttestationSlot=4858996 nextProposalSlot=-1 syncCommitteeDuties=none head=239dd906:4858983

To Reproduce
Steps to reproduce the behavior:

  1. Ubuntu 20.04, Nimbus 22.10.0, besu/v22.10.0-RC1/linux-x86_64/openjdk-java-11
  2. Relevant log lines:
    Oct 07 04:35:38 hostname nimbus_beacon_node[748]: ERR 2022-10-07 04:35:38.064-04:00 Corrupted deposits history detected topics="eth1" ourDepositsCount=450876 taretDepositsCount=450876 ourDepositsRoot=cde455fc targetDepositsRoot=ace544b6

I am not certain this is truly a defect, as I don't see any adverse result such as missed attestations, etc.
@mratsim
Copy link
Contributor

mratsim commented Oct 7, 2022

This sounds related to hyperledger/besu#4495

Workaround while investigation continues:
stop besu
delete the caches/ folder under besu's data directory
start besu
restart teku

@jfblaine
Copy link
Author

jfblaine commented Oct 7, 2022

Thanks for the pointer. I deleted the ./caches dir. Will keep an eye on things and report back.

@jfblaine
Copy link
Author

jfblaine commented Oct 7, 2022

Deleting the besu ./caches folder seemed to do the trick:

image

Apologies for spamming nimbus issues with this.

@jfblaine jfblaine closed this as completed Oct 7, 2022
@ibhagwan ibhagwan mentioned this issue Feb 25, 2023
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@jfblaine @mratsim