Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Missing trie node after initial snap sync #29436

Open
rjl493456442 opened this issue Apr 2, 2024 · 2 comments
Open

Missing trie node after initial snap sync #29436

rjl493456442 opened this issue Apr 2, 2024 · 2 comments
Labels

Comments

@rjl493456442
Copy link
Member

rjl493456442 commented Apr 2, 2024

The original report is from #29417

Geth:

  • scheme: hash
  • version: 1.13.14-stable

reports error like this:

Apr 01 18:04:59 aviad-SEi geth[69608]: ERROR[04-01|18:04:59.141] Unhandled trie error in Trie.Prove       err="missing trie node 67120b27ad8a3045df012b9e4e2daae8ed047f5e52bc7800356b6b9cd633709c (path 0000010a060a0b) <nil>"

The error means an account trie node at position []byte{0x0, 0x0, 0x1, 0xa, 0x6, 0xa, 0xb} is missing.

The correct node blob should be 0xf8669d35fce6efd2272cdb9ea2eacce639fab06c112ac8be435a182bae332defb846f8440180a0bbd13d463ce2111713bb0d6ad117b0ba991bc32336d501e872703c4ad6bc3c90a0905982d3bc98b3ad75d5ca7a92446cc8923fae39a2352245000f9d3e9dc0f3ee, can be decoded as a shortNode:

[
  35fce6efd2272cdb9ea2eacce639fab06c112ac8be435a182bae332def,
  f8440180a0bbd13d463ce2111713bb0d6ad117b0ba991bc32336d501e872703c4ad6bc3c90a0905982d3bc98b3ad75d5ca7a92446cc8923fae39a2352245000f9d3e9dc0f3ee,
]

After inspecting the history (from block 19400000, roughly Mar-10-2024), the account mutation with address-hash prefix 001a6a is found as below:

./geth db inspect-history --datadir . --start 19400000 --prefix 001a6a --datadir ./geth_data

Account history:

        blockrange: [#19400000-#19600717]
#19541282
                001a6aefc178724fdc1d7f0f7b655fca3dfb9d5ae36b2425d9efe98c1d0e7126 - (address: 3b675ffd72a3be289d1cb3e8cceb0417505d5fcd): 0xcd820121871ad7eef6b7da568080

It's a sibling of missing node mutated on Mar-29-2024

@rjl493456442
Copy link
Member Author

Another similar report is from Geth-discord server:

Geth:

scheme: hash
version: unknown

Mar 19 11:25:48 OrangePi5P geth[15795]: INFO [03-19|11:25:48.482] Resuming state snapshot generation       root=f3b387..c43e63 in=1df820..0b7a24 at=290dec..f3e563 accounts=31,432,500
slots=202,708,968             storage=15.88GiB   dangling=0 elapsed=4h45m37.902s eta=35h54m15.615s
Mar 19 11:25:48 OrangePi5P geth[15795]: INFO [03-19|11:25:48.549] Imported new potential chain segment     number=19,468,484 hash=90d01d..445b7a blocks=1         txs=174         mgas=12.808  elapsed=241.250ms    mgasps=53.091  snapdiffs=354.57KiB triedirty=802.61MiB
Mar 19 11:25:48 OrangePi5P geth[15795]: INFO [03-19|11:25:48.641] Chain head was updated                   number=19,468,484 hash=90d01d..445b7a root=73dd6f..e436c2 elapsed=11.101605ms
Mar 19 11:25:48 OrangePi5P geth[15795]: ERROR[03-19|11:25:48.804] Unhandled trie error in Trie.Prove       err="missing trie node e6fdf16ed01e73368d8ee4f80b9f6d3f54fe7eb335397503de6d0bb3bc5e4ec1 (path 010d0f0a060e) <nil>"
Mar 19 11:25:48 OrangePi5P geth[15795]: ERROR[03-19|11:25:48.811] State snapshotter failed to iterate trie err="missing trie node e6fdf16ed01e73368d8ee4f80b9f6d3f54fe7eb335397503de6d0bb3bc5e4ec1 (path 010d0f0a060e) <nil>"

The missing node is a fullNode in account trie

0xf90111a08e0ba3b973c207759cc214d0cdab06c9955b580dcc1eb6e68590b87a277fb09c808080a0f252843b56a5a339c7413cd233b82ff8bf034e6f8bfda7e77558671e318dc8afa0c598146e383e0aebf94fa73900bfeb9a0731d1fb52c58c365ca0528f08160f20808080a093494ce0ff9969d9c3ba3d965118557bd376a58877aeac1918ba466d98c009b4a05ad20e32eb2c575bbd6ad27094223739d82515132c7ec92b69fc8f1d44da62b9a01d7e731a5d22d8db4e107aeb749bf25e985d1a57b5e658e287d7e279056b5b8da0f0b1e09ee6d8af2ae08fd8cf698ff982e2387408d637d3f468d132de0dec86c080a0a6f35629bb9ef5fe1c36354098af4c4cbef0975779b8da89b3bea9013f899f8f8080

[
  8e0ba3b973c207759cc214d0cdab06c9955b580dcc1eb6e68590b87a277fb09c,
  "",
  "",
  "",
  f252843b56a5a339c7413cd233b82ff8bf034e6f8bfda7e77558671e318dc8af,
  c598146e383e0aebf94fa73900bfeb9a0731d1fb52c58c365ca0528f08160f20,
  "",
  "",
  "",
  93494ce0ff9969d9c3ba3d965118557bd376a58877aeac1918ba466d98c009b4,
  5ad20e32eb2c575bbd6ad27094223739d82515132c7ec92b69fc8f1d44da62b9,
  1d7e731a5d22d8db4e107aeb749bf25e985d1a57b5e658e287d7e279056b5b8d,
  f0b1e09ee6d8af2ae08fd8cf698ff982e2387408d637d3f468d132de0dec86c0,
  "",
  a6f35629bb9ef5fe1c36354098af4c4cbef0975779b8da89b3bea9013f899f8f,
  "",
  "",
]

After inspecting the history (from block 19400000, roughly Mar-10-2024), the account mutation with address-hash prefix 1dfa6 is found as below.

Account history:

        blockrange: [#19400000-#19600717]
#19408862
                1dfa67ab98fa17e1e07333e8775afd82eb67507342d65166e4ea61b43f22101f - (address: eedf737b1ae162bdc6c3c1f9d9710641cbdf9c9c): 0xcc8088074f490ca01800008080
#19455625
                1dfa69bc89f51034e41e3e58be59d18bd13e9ff327331b0017e88c97cdddc3f8 - (address: 0b583b9a448e44bf7c6975d207f8f3e4d43778f4):
#19484010
                1dfa633edfe0436f98846530f6e939f915628ab4854a5854238fade9c4bddc59 - (address: 440bdd191dbdf7cadef6fd9d901f24782dab77ae):
#19484034
                1dfa633edfe0436f98846530f6e939f915628ab4854a5854238fade9c4bddc59 - (address: 440bdd191dbdf7cadef6fd9d901f24782dab77ae): 0xcb8087c50554634433948080
#19484163
                1dfa63df42783eef34d7e4b15b128bc62f3452a91ea3b58e93cad290bac2e4cf - (address: bbb82009b3af274c7b7f392f3dfa326c60950475): 0xcc22880137c9f8941b7ab78080
#19491753
                1dfa60e6344cd434a8df6e181cd25e9cc84e567e7d554085a6598196bf32f00d - (address: cc6952a0108d0d5504457a7aa5383244c0046bd3): 0xcb01870876569eb15c6b8080
#19512048
                1dfa633edfe0436f98846530f6e939f915628ab4854a5854238fade9c4bddc59 - (address: 440bdd191dbdf7cadef6fd9d901f24782dab77ae): 0xcb0187012275f4d82a608080
#19519217
                1dfa633edfe0436f98846530f6e939f915628ab4854a5854238fade9c4bddc59 - (address: 440bdd191dbdf7cadef6fd9d901f24782dab77ae): 0xcc018802c7ad66afec2a608080
#19569204
                1dfa620c24104c229bb057896297422a1b27a8d4167ef6a8cd70dbff397d3ab6 - (address: a620260d42315a2532852339f4b7ae4db5e40407): 0xcb0887879f2b906f1f5e8080
#19569219
                1dfa620c24104c229bb057896297422a1b27a8d4167ef6a8cd70dbff397d3ab6 - (address: a620260d42315a2532852339f4b7ae4db5e40407): 0xcb0887e395f0d4ffeb5e8080
#19569305
                1dfa620c24104c229bb057896297422a1b27a8d4167ef6a8cd70dbff397d3ab6 - (address: a620260d42315a2532852339f4b7ae4db5e40407): 0xcc0888b54cdd105513eb5e8080
#19580498
                1dfa6191a48f0a3c1ac6ad696c0f7a8bd16573865316d709e499452346a75c98 - (address: e927064b56920f9edf3930d3f23fdfabc18d8290):
#19587242
                1dfa6385c300e5310e3baec66ed016e0f97c4a765e1aa3a8350f8078de2b081e - (address: 87ba13643bca4c44c110066150242b216ecaf53d): 0xcc808803358f939ad408008080
#19587270
                1dfa6385c300e5310e3baec66ed016e0f97c4a765e1aa3a8350f8078de2b081e - (address: 87ba13643bca4c44c110066150242b216ecaf53d): 0xcc0188032a89cdb065fc348080

Several siblings are mutated, especially the one in block 19455625 which is on Mar-17-2024

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants
@rjl493456442 and others