Investigate a broken snapshot after archive sync

[AlexeyAkhunov]

After having performed archive sync, I transmitted the database (around 500Gb) to another computer and tried to resume sync from it.
However, the sync very soon failed because the resolution of one of the contracts returned a wrong storageRoot.
The success of transmission has been checked by performing SHA256 on both files, and the results matched.
The plan is to compare storageRoot for all contracts with the computed values, then choose the smallest contract where the mismatch happens, and attempt to find the cause for the mismatch.

[AlexeyAkhunov]

Snapshot verification showed that the file itself is good, all storageRoots match what the resolvers for each contract's storage return

[AlexeyAkhunov]

INFO [10-16|17:11:38.153] Block synchronisation started 
INFO [10-16|17:11:40.194] Inserting chain                          start=8739813 end=8739814
INFO [10-16|17:11:40.389] Creating IntraBlockState from latest state block=8739812
INFO [10-16|17:22:18.081] Rebuilt top of account trie and verified root hash="<8588eec79ee1be47bdcdcb26a7a0eb6866bf3b8951603f8a34b88deb4bc92969> "
INFO [10-16|17:22:18.081] Memory after rebuild                     nodes=69905 alloc=117750 sys=280990 numGC=268
INFO [10-16|17:22:18.081] Creation complete. 
INFO [10-16|17:24:59.187] Imported new chain segment               blocks=1 txs=117 mgas=4.689 elapsed=13m19.020s mgasps=0.006 number=8739813 hash=ae611a…6192d0 batch=833 age=2d2h37m
INFO [10-16|17:25:00.052] Memory                                   nodes=70846 alloc=149485 sys=349388 numGC=306
INFO [10-16|17:25:00.053] Database                                 size=0 written=10825729
ERROR[10-16|17:25:52.902] 
########## BAD BLOCK #########
Chain config: {ChainID: 1 Homestead: 1150000 DAO: 1920000 DAOSupport: true EIP150: 2463000 EIP155: 2675000 EIP158: 2675000 EIP2027: <nil> Byzantium: 4370000 Constantinople: 7280000  ConstantinopleFix: 7280000 Engine: ethash}

Number: 8739814
Hash: 0x748c96fae5304677148f6bc8c2c77d1f279219a3aa7d28dbb3796358f26183c0

... LOTS OF LINES HERE OMITTED ...

Error: mismatching hash: <01ec34543148889bce381afb1e958ef5491ee1cce28cbd2500f81a2c22e20157>  086da6357e40909c2e75702b358980483f63d44b4f3c59d785eda7942f507879 for prefix 8679e8eda65bd257638cf8cf09b8238888947cc3c0bea2aa2cc3f1c4ac7a3002ffffffffffffffff, resolveHex 0001040003060103080d060f02040b080e0d0702040409070a0c020a01050800010702060801060c0d0a0404010a020b0603060c070607090d0a0f000a0c0c0d10, resolvePos 0

[AlexeyAkhunov]

It turned out that the snapshot was broken, and it was not picked up because the verification code was incorrect. Now the verification code is fixed, there are a few contracts for which the storage Roots do not match:

adf717bc28f8c3a19a1693d1977bacc3d4ebd90ee8fd1d69acabda69a99f047f: mismatching hash: <a5756b61dfe2ba0d3177fbb30c50e6b1027f1a9bd84e25fb68142d73c33116c8>  d06b7c361192cf7c12bff5dcd27b5226b9f564535aee7d9cfbb36ad75f636c5a for prefix adf717bc28f8c3a19a1693d1977bacc3d4ebd90ee8fd1d69acabda69a99f047fffffffffffffffff, resolveHex , resolvePos 0
d3ced05b8829a08a9ce303e585c3f60bed8c4b772b0dc5aa89528f6b9af467ab: mismatching hash: <d8f56c053e6fa9251ad530bfc02f562ebbac51531e64e7026d20f6b6a4406221>  83fe29804b7a4d11e928147a91b21ffdf3e56d644351ed4da139b3aa66ec33fa for prefix d3ced05b8829a08a9ce303e585c3f60bed8c4b772b0dc5aa89528f6b9af467abffffffffffffffff, resolveHex , resolvePos 0
9d900b013d41cd2b54a0d7e8247c4a5674846f6f85bf29061eee5cb6bc88ccc5: mismatching hash: <1c11ad1a0e60155bc208ca5db52988537405dde810502e2757c17ddcfd7748be>  e88ce5c910487688803537fbb9ec2a5a6af916d893884ad8d2b5f0f01ae3c763 for prefix 9d900b013d41cd2b54a0d7e8247c4a5674846f6f85bf29061eee5cb6bc88ccc5ffffffffffffffff, resolveHex , resolvePos 0
0e6be6f355600ddfa8b7ec773018ee584c1014980a3448d40fc338427e2b158b: mismatching hash: <870d6bc9248dfc65d2ab579300cedc7fc5d53184f371220a1ccca5239cc84bd1>  d9f32b89d7cdac5fb79090273da7584856dfd8324c40824b3946af40f86b993c for prefix 0e6be6f355600ddfa8b7ec773018ee584c1014980a3448d40fc338427e2b158bffffffffffffffff, resolveHex , resolvePos 0
a47beea8510ae55482e7bb3db7dabf6e93e508da86ae93bf3d7f7f76b3cd81d8: mismatching hash: <0528464146349a9b2ac9e322e614552d8c327d393914d919950e54cd2c6f5d1f>  b48580c9abe2759b50254066851e5cc3e185c0a2aa07acdf7652bd928364d743 for prefix a47beea8510ae55482e7bb3db7dabf6e93e508da86ae93bf3d7f7f76b3cd81d8ffffffffffffffff, resolveHex , resolvePos 0
9669915f46890a68f2005cc2d23c52ad1a62d5c70ac5de43ae87b8c8dab0a989: mismatching hash: <998559aaaeca3de8efbd938c9fd06cd8a9122c77c8fec13fc6572414a67f17e1>  1261d622c49c63e10c3f605673a5a5f4739da85196d769c54ec4f3ae602ad621 for prefix 9669915f46890a68f2005cc2d23c52ad1a62d5c70ac5de43ae87b8c8dab0a989ffffffffffffffff, resolveHex , resolvePos 0
8679e8eda65bd257638cf8cf09b8238888947cc3c0bea2aa2cc3f1c4ac7a3002: mismatching hash: <01ec34543148889bce381afb1e958ef5491ee1cce28cbd2500f81a2c22e20157>  086da6357e40909c2e75702b358980483f63d44b4f3c59d785eda7942f507879 for prefix 8679e8eda65bd257638cf8cf09b8238888947cc3c0bea2aa2cc3f1c4ac7a3002ffffffffffffffff, resolveHex , resolvePos 0
23c7b3537befb27bab42c2588ab80cf00423aad07002fe9be4205066237b5479: mismatching hash: <a089a51ca8522351684b012009e6afcadf360ffe9ee09ac408aae6c160df9ed0>  2954dc541773047b9050c6acf0495e85e7edab62f09567164ff5fc26d4ee54f1 for prefix 23c7b3537befb27bab42c2588ab80cf00423aad07002fe9be4205066237b5479ffffffffffffffff, resolveHex , resolvePos 0
af0267842c5acc036c253cfa4305acb0f295f9e0c3523a55906f7d1ce319a227: mismatching hash: <5de4ed14713caad49b627b9bf5bb3d76f809f2e5f17aebce3ae9826557f11952>  fb827d4644a9034df4e28b5ea6ad281584d221462a6bf0bc595878ee8d97af23 for prefix af0267842c5acc036c253cfa4305acb0f295f9e0c3523a55906f7d1ce319a227ffffffffffffffff, resolveHex , resolvePos 0
741e6376a392932114d0971e88e4e6c094ca00128da3c1d0cb11061663ba921f: mismatching hash: <0546e5a36e7a2d6756bebc5b962379287311c00631a6856300a7bd5b22ac8a67>  56860a75086bb7398234cf02912567a810830d6621dbfbf841699606bf7f19a4 for prefix 741e6376a392932114d0971e88e4e6c094ca00128da3c1d0cb11061663ba921fffffffffffffffff, resolveHex , resolvePos 0
93f37d58cf87ec06994a7095fddb38780e4da07b7cc978d6610431854c69e723: mismatching hash: <05045b61e11d0632a4e66cf397929683a6cd793cb5e31857fdc516237fe96409>  0bb9446403d5bab6379824a0d3edbb3ad88c12160f2294386860a24570c9c547 for prefix 93f37d58cf87ec06994a7095fddb38780e4da07b7cc978d6610431854c69e723ffffffffffffffff, resolveHex , resolvePos 0
3e5afb8406f3d16c691fc44c4278c8e20ed1b3ad71eeda2d03dab65db7a57fb9: mismatching hash: <48f25321a97e4f62969bdc83b392d17749e2f7ab95905f24b869bc6f89b1f967>  09680281266ebb96fa20f52da7017c3d0b04624fd8dbc2dce5c47836fb6c8bab for prefix 3e5afb8406f3d16c691fc44c4278c8e20ed1b3ad71eeda2d03dab65db7a57fb9ffffffffffffffff, resolveHex , resolvePos 0
29b01fcf1896e29e24ca39d48ac3e1be1170e0b52993a96f3366288c861678a1: mismatching hash: <573f82555930106bceec004006a9e2a55adcb0d638a4f6346c10e20e5a0a914c>  bc344206f0755aa6ed4fd23aaf67e458bac228117c52084f7b7854e009faeb28 for prefix 29b01fcf1896e29e24ca39d48ac3e1be1170e0b52993a96f3366288c861678a1ffffffffffffffff, resolveHex , resolvePos 0

[AlexeyAkhunov]

The smallest contract with the mismatched storage root seems to be:

80b1dc852f3aeef243ffdd2d47c7435b21284dcb

corresponding to the hash

a47beea8510ae55482e7bb3db7dabf6e93e508da86ae93bf3d7f7f76b3cd81d8

[AlexeyAkhunov]

Compared storage of contract 80b1dc852f3aeef243ffdd2d47c7435b21284dcb at the block 8739812 with the correct one (got from one of the Infura's nodes).
Out of 385 entries, there 1 difference. The correct value is:

      "0xcb328570877b6c26f437a5edbfb039f5189729fa4033ad68b0a4a885a64262f1": {
        "key": "0xcde45d1dc8ae2ef209d9ea95691065b130732d6ca387a8cfd8fead9eab90edb9",
        "value": "0x000000000000000000000000000000000000000000001b7ad8861912a9680000"
      },

the incorrect value in the snapshot is:

0000000000000000000000000000000000000000008a1b7ad8861912a9680000

[AlexeyAkhunov]

Binary search on the transaction where the value 1b7ad8861912a9680000 has been set.

After block 8608319: 30a7db4dfa5d9fe80000
After block 8674065: 30a7db4dfa5d9fe80000
After block 8706938: 30a7db4dfa5d9fe80000
After block 8723375: 30a7db4dfa5d9fe80000
After block 8731593: 1b7ad8861912a9680000
After block 8727484: 1b7ad8861912a9680000
After block 8725429: 30a7db4dfa5d9fe80000
After block 8726456: 30a7db4dfa5d9fe80000
After block 8726970: 30a7db4dfa5d9fe80000
After block 8727227: 1b7ad8861912a9680000
After block 8727098: 1b7ad8861912a9680000
After block 8727034: 30a7db4dfa5d9fe80000
After block 8727066: 1b7ad8861912a9680000
After block 8727050: 30a7db4dfa5d9fe80000
After block 8727058: 1b7ad8861912a9680000
After block 8727054: 1b7ad8861912a9680000
After block 8727052: 30a7db4dfa5d9fe80000
After block 8727053: 1b7ad8861912a9680000

So the modification happens in the block 8727053. It has 91 transactions. Binary search on transactions:
After transaction 90: 1b7ad8861912a9680000
After transaction 45: 30a7db4dfa5d9fe80000
After transaction 67: 1b7ad8861912a9680000
After transaction 56: 1b7ad8861912a9680000
After transaction 50: 1b7ad8861912a9680000
After transaction 47: 1b7ad8861912a9680000
After transaction 46: 1b7ad8861912a9680000

So it happens in the transaction number 46. Transaction ID is 0x6737962c4d8b8fb1592ed4874e7e5235f0d0d5fa5f5fb11c7cd9c25a7cdb80e1

[AlexeyAkhunov]

Crucial observation (I think). The incorrect value 8a1b7ad8861912a9680000 is the RLP-encoding of the correct value 1b7ad8861912a9680000

[AlexeyAkhunov]

Conclusion is that the rewinding the blocks that contain storage modifications ends up adding RLP-wrapping to the storage values

[AlexeyAkhunov]

Looks like the number of mismatched storage items is not large:

Contract: 000007222caeb29694719e804b24fb3ee5116a8a
storage entries: 2175
keyHash 3252a0321cdfceb12940e976577c177db7a91f8fe8142649d28b79d84cb670a9, value 83841253, expected 841253
keyHash 98f2aadd3c8c619c5a8f431589635778470fa8ecd45508dd6076b7ba267647cf, value 83841253, expected 841253
keyHash c3d58b3acc63710a65cdf2b68e4726f8534b32ec79539480764f09bef9e04d2a, value 83841253, expected 841253
keyHash f84ba31595c631bc0cfac8ce593c6d55d8fd44ac21b5467976ba1b36ed449ac2, value 83841244, expected 841244
Contract: 06a6a7af298129e3a2ab396c9c06f91d3c54aba8
storage entries: 1677349
keyHash 13f65afe62b726cb7c20f359e96bc08f908a9567a27c5e04a691dca0dded0bcb, value 81ee, expected ee
keyHash 1736b7647a1af71e886358b8144f405b1e2d138e202a18d8ea234abaf787ff07, value 81e7, expected e7
keyHash 1849c8a46be65b1ca07f2d5052252e7ac6e37fc2eab1aa343bbb13f99558a973, value 81ea, expected ea
keyHash 2239500e8036ed430a023622a5914a66e5de8906128a8934d9dfbee6521410ce, value 81ec, expected ec
keyHash 37cade200af8e136c2b00b08a240e3ccf2c652694ce0a48c39d0e4e14b560f1b, value 81e9, expected e9
keyHash 39d8a5bf13dd57898d95a6b99b6c34f1a1895a90ad9f4ba5c62d2c4f05cb0894, value 81e9, expected e9
keyHash 4b3429ed1a233b6cd0901fcfacfe95220cc5ac70fcffb8ca2a99389a4553a9e8, value 81eb, expected eb
keyHash 6ddbd5db32d506d23335fc8c97bfb92634b3bfeaa860e08d8b953818a1acb61c, value 81e9, expected e9
keyHash 70f1b9973e0fcfbbbc1268ff1f426e50b96c7a1eddae950e5eb1e3bb2e4ec964, value 81e9, expected e9
keyHash 84ab02057b989e14ad2f07f5e3f1a405b6c91f1072ac1f6c1d826c276c1123e7, value 81ea, expected ea
keyHash 8cdab1d6541c308587f777f8b1254a3bc70c13e5a19e844f9ba4f907d294b203, value 81e6, expected e6
keyHash d0abe000c66d158888823719cf764ad9b6ba4122a453e7fbb288e95615ab5d95, value 81ec, expected ec
keyHash d36e7206ac78954a3c554a8ec792515be9c97923036b9b58981136955646d15a, value 81ec, expected ec
keyHash dba457af9406ffe29a1cd0954962c2826d855b9113f1808f40d72aefc9bd5235, value 81e9, expected e9
keyHash dbbe0389414d4dde2e1c3e38d8cd974e585f814ddc06d05bb4cf9beaa5a8a46c, value 81ec, expected ec
keyHash e9afb603a074b44115c9870c85d6e35814d596b68a9b5a227b45b613531b0550, value 81e7, expected e7
Contract: 1a95b271b0535d15fa49932daba31ba612b52946
storage entries: 10025
keyHash f639a988aa288164084e799f8c5783563dbcf4085c7c7ef1c5c9b9f20c18ef7c, value 8602c22a8c9800, expected 02c22a8c9800
Contract: 39755357759ce0d7f32dc8dc45414cca409ae24e
storage entries: 1438606
keyHash 388aaad5c2f8a600d09a841deba0d8401ab3ab28b9db9964941f700d0d627c70, value 8307d883, expected 07d883
keyHash 762aa5bf19e77a6ed53cfcf3b0bd88c36a8a91f7aefdbd4d8357f2a7c028c4ac, value 8307d206, expected 07d206
keyHash a95e0894f0c49e9155814f0a6105ca886248efe770f1739d90316d6397b799a3, value 8307d408, expected 07d408
Contract: 4c9d5672ae33522240532206ab45508116daf263
storage entries: 261918
keyHash 6e1540171b6c0c960b71a7020d9f60077f6af931a8bbf590da0223dacf75c7af, value 8c8bb1bf612f95b26364800000, expected 8bb1bf612f95b26364800000
keyHash 7010b32ac6eb0890047b0abb25b17ec6002b610da29f149d8d6636d2622f8e52, value 8b1174194abeb0c92b800000, expected 1174194abeb0c92b800000
keyHash f3f7a9fe364faab93b216da50a3214154f22a0a2b415b23a84c8169e8b636ee3, value 8c8bc618a1dba8306c43800000, expected 8bc618a1dba8306c43800000
Contract: 5b135d7e2774c801a73208f258123d7623e07784
storage entries: 37655
keyHash 7c6751b6da0debdc9713a18ee40b3665416be6a367ca60c83777aedeccb50238, value 88d02ab486cedc0000, expected d02ab486cedc0000
Contract: 80b1dc852f3aeef243ffdd2d47c7435b21284dcb
storage entries: 385
keyHash cb328570877b6c26f437a5edbfb039f5189729fa4033ad68b0a4a885a64262f1, value 8a1b7ad8861912a9680000, expected 1b7ad8861912a9680000
Contract: ae9b8e05c22bae74d1e8db82c4af122b18050bd4
storage entries: 179554
keyHash 44a916fd7c85486c43f4241d397706147dce99fde9a9efb117de4a50ec0a1e83, value 85230000000d, expected 230000000d
Contract: c02aaa39b223fe8d0a0e5c4f27ead9083c756cc2
storage entries: 140779
keyHash cb328570877b6c26f437a5edbfb039f5189729fa4033ad68b0a4a885a64262f1, value 84b2d05e00, expected b2d05e00
Contract: df0960778c6e6597f197ed9a25f12f5d971da86c
storage entries: 565
Contract: e3818504c1b32bf1557b16c238b2e01fd3149c17
storage entries: 23578
keyHash 9b461be3d182352e15cafb2641b0bdf05847f6e33f2d2a4e908f8e1eab54ed2e, value 890217fbd1e2569913ab, expected 0217fbd1e2569913ab
Contract: f936aa9e1f22c915abf4a66a5a6e94eb8716ba5e
storage entries: 3841
keyHash 18a91ebbc07e1dedeb89ebbcca26c9080d1f49cc33899ef58877a6a154ff5b79, value 8b010000000000000000014c, expected 010000000000000000014c
keyHash 2ef6af70e92d2d867ece55e3595e57b7c55969f8748eb543bbfd7378b4ef22f0, value 9d10000000000000000000000000000000000000000000000000000002d0, expected 10000000000000000000000000000000000000000000000000000002d0
keyHash 3a6dffc6b0009975c8d87ab00aed4a329f2066b9c78a97a49d76646b87453e57, value 9d2d000000000000002d000000050000002d000000000000000000000000, expected 2d000000000000002d000000050000002d000000000000000000000000
keyHash 42fc602cec0bee860f44f22d2ec51db4b809c78188cc8854ceb34267d8e6da9a, value 9d09000000000000000000000000000000000000000000000000000002f8, expected 09000000000000000000000000000000000000000000000000000002f8
keyHash cae250a8bfccc77b5fb670882d68e373ecf70f5219f072ad5d5679558a437f2a, value 9d050000000000000000000000000000000000000000000000000000015e, expected 050000000000000000000000000000000000000000000000000000015e
keyHash eb3c1d0ba3d08d2fa2593def039f98c5f5dcfe6ee82a583139cc2e1423674e42, value 9d1300000000000000000000000000000000000000000000000000000000, expected 1300000000000000000000000000000000000000000000000000000000
Contract: fb80bfa19cae9e00f28b0f7e1023109deeb10483
storage entries: 370
keyHash 73f9ecab0298f240c30af2217d12443552eabc129e59a7de4283dc42c473e38e, value 9b855a8effffffffffffd83b0000000000855aaeffffffffffffd83b, expected 855a8effffffffffffd83b0000000000855aaeffffffffffffd83b