Error: Failed to update deposit cache

[macfarla]

Reported in Discord

Lighthouse v3.1.2-01e84b7
Besu v22.7.5
Oct 07 03:44:05.686 ERRO Error updating deposit contract cache error: Failed to update deposit cache: "FailedToInsertDeposit(NonConsecutive { log_index: 450832, expected: 450831 })", retry_millis: 60000, service: deposit_contract_rpc

We (Protocols) are also seeing this on some Besu/Teku nodes that we run. at least one is running Besu 22.7.2

[macfarla]

Workaround while investigation continues:
stop besu
delete the caches/ folder under besu's data directory
start besu
restart teku

[macfarla]

for additional context, LH requests the deposit contract logs in 1000 block ranges. The error implies that the logs in the given range are not in strictly increasing order. From past experience, this usually means that the EL is missing a few logs in that range

[macfarla]

teku nodes with this issue showing Encountered a problem retrieving deposit events from eth1 endpoint: Expected next deposit at index 450831, but got 450832

[macfarla]

Restarting besu (and deleting the caches/ dir) and restarting teku/lighthouse has resolved the problem for everyone I've seen so far.

[macfarla]

restarted with passing --eth1-purge-cache to lighthouse and errors have disappeared after lighthouse rebuilt it's cache

[macfarla]

looking at etherscan, there seems to be an indexing anomaly there too

https://etherscan.io/tx/0x16c01fc1e96db9aae83f96ef2e23d26ef99806f47c1fd64c2982eacd1234d151#eventlog 48
https://etherscan.io/tx/0xf6646ac0e7515b45771612bcc74d417bf6a4796641ec5f3972d455cc7a58f5d6#eventlog 47
https://etherscan.io/tx/0xab0f4c9b663567426d024eeb353bfe1bb09a3c443c264a6dda70fc0f39ffb295#eventlog 46
https://etherscan.io/tx/0xfc0c8dc0103a864fd919626f83f8345fa76bde90ac45e77fd43712229741928a#eventlog 45
https://etherscan.io/tx/0xdfb91735f9c03e6a82ec54a9b15b54e532bb731e8b8debfcc3254f6952dd9fa3#eventlog 44
https://etherscan.io/tx/0x718504398817a451774f02fd92bc97b3b3166506b160a8354ac1366e76623660#eventlog 43 
https://etherscan.io/tx/0xb6bf4af2e9bb7b2acebf3949c2ccc6ae26648078dc4cc1ec9a0696bc5b7dfdb6#eventlog 42
https://etherscan.io/tx/0x15287968ba4ef101140ade83cea6ae1d82a1cd73faad997955bcab385f3ef3f1#eventlog 41
https://etherscan.io/tx/0xaaa514e59d17346cd317b6ec6cc36a75717b2223e6c05a9e53fecc4088ef6a03#eventlog 40
https://etherscan.io/tx/0x7c2885e2b424be1f5875bbe875c6f212cbe4f330d426cfa804df3dd8fc40d24b#eventlog 3B
<the one we want 0FE106 should be in here> 
https://etherscan.io/tx/0xff84d06dc1960c5b706e59a5f600ae3c6c4625933747b1ff3d858b820a1d1c04#eventlog 0E
https://etherscan.io/tx/0xde9155ee3d49831d94b95718ef6e345b535c2e035c81cfaea617b4587fa585c5#eventlog 0D
https://etherscan.io/tx/0x9e92276b4c92a9eedc570ddadf0c05a2f219165375801e0101f7779774cfca0f#eventlog 0CE1060000000000
https://etherscan.io/tx/0x4a15c42a3b96654bfb6ae9d96c6b0328ee57201d01afb669c68088a602dd9523#eventlog 0BE1060000000000
https://etherscan.io/tx/0x47b39ec56a18c3f2c404f21cf22c8cee2b5bfab048ebd4ef3887a86c9ae51abb#eventlog 0AE1060000000000
https://etherscan.io/tx/0x6895df8f4f144181c9aa77c7c411797c25d96bb36365f3083c118565e3ea6c15#eventlog 08E1060000000000
https://etherscan.io/tx/0x89a7c8ebbb15bdf7a328dc19658160024ffbc2fed949c120dc27414fb4895d1d#eventlog 07E1060000000000

[macfarla]

450831 is 06E10F in hex and 0FE106 in little endian - which matches the "index" of the log in etherscan

[jflo]

The current working theory is that when a re-org happens the logs bloom cache is not updated correctly. We are attempting to reproduce in test code.

[lincoln2020]

I'm having the same problem with Lighthouse 3.1.2, Besu 22.7.4. I know how to stop/start Besu, but how exactly do you "delete the caches/ folder under besu's data directory" with the terminal only? Thanks

[macfarla]

@lincoln2020 sorry for late reply
if you look under besu's data folder you will see a caches dir

➜  besu1 ✗ ll data
total 16
-rw-r--r--   1 sallymacfarlane  staff    13B  1 Aug 12:12 DATABASE_METADATA.json
drwxr-xr-x  18 sallymacfarlane  staff   576B 12 Oct 12:10 caches
drwxr-xr-x  60 sallymacfarlane  staff   1.9K 12 Oct 12:10 database
drwxr-xr-x   3 sallymacfarlane  staff    96B  8 Sep 14:54 fastsync
-rw-r--r--   1 sallymacfarlane  staff   159B 12 Oct 11:40 static-nodes.json

command to delete would be
rm -rf data/caches