ar_header_sync_block_tx_root_mismatch

[ThomasBlock]

On one of my nodes the following warning is issued 10 times per second on repeat ( for different blocks ). The node has synced 99,5% of the block headers and new headers are added without problem. But the difference between arweave_storage_blocks_stored and arweave_block_height is always 5057.

2022-08-16T15:29:08.613402+02:00 [warning] /root/arweave/apps/arweave/src/ar_header_sync.erl:484 event: ar_header_sync_block_tx_root_mismatch, block: <<"y0vlt5dHDXB22pf_nqlHlDJfZcAhwadeuFOHTdl_XDeloQTj7h7mGuj76eRCGRvD">> 2022-08-16T15:29:08.847703+02:00 [warning] /root/arweave/apps/arweave/src/ar_header_sync.erl:484 event: ar_header_sync_block_tx_root_mismatch, block: <<"ZicWdamH_9Ii6OmQEoHNo2ygFjTpE1xnlMjzGoBmEMIhQaMWc3DOgwZg82pvr9YZ">> 2022-08-16T15:29:08.966679+02:00 [warning] /root/arweave/apps/arweave/src/ar_header_sync.erl:484 event: ar_header_sync_block_tx_root_mismatch, block: <<"WURgtmYEshWP6fzIgI8_a8oQaAgr7cNPeYQlqaOqzFFSb4w1sOcGqVo6jmN7U3zq">> 2022-08-16T15:29:09.021672+02:00 [warning] /root/arweave/apps/arweave/src/ar_header_sync.erl:484 event: ar_header_sync_block_tx_root_mismatch, block: <<"LfJoAWmGcXh9eY4qBsmramtmqmWWP-3JOU46gyacMrumM742wAbXEY-Ce3WQp3yl">>

So there seems to be a problem with the old headers?.. I have enough storage, bandwidth and peers. I tried to change parameters ( "header_sync_jobs":1 and also "header_sync_jobs":10. i tried "sync_jobs":0 ) but even after 48 hours this does not change a little bit.

I probably could delete the data folder and resync the weave. but is there a way to repair it? rocksdb is still a mystery for me.. could i delete files or edit the db somehow?

[ldmberman]

Hi, I think, you've run into a bug fixed here - could you try this branch?

[ThomasBlock]

So i compiled the fix and let it run for 10 hours.. high cpu and ram usage as usual... but i can not see any difference.. maybe these logs are new.. but probably not related:

2022-08-18T20:36:32.072919+02:00 [warning] /root/arweave/apps/arweave/src/ar_data_sync.erl:1135 event: invalidating_bad_data_record, type: 4, range_start: 94966040010998, range_end: 94966040273142

2022-08-18T21:51:06.499926+02:00 [warning] /root/arweave/apps/arweave/src/ar_data_sync.erl:1135 event: invalidating_bad_data_record, type: 2, range_start: 94969955393782, range_end: 94969955655926

2022-08-18T17:36:20.766286+02:00 [error] : Ranch listener ar_http_iface_listener, connection process <0.17525.105>, stream 1 had its request process <0.19372.105> exit with reason {case_clause,eof} and stacktrace [{ar_storage,read_block_from_file,1,[{file,"/root/arweave/apps/arweave/src/ar_storage.erl"},{line,811}]},{ar_http_iface_middleware,handle_get_block2,3,[{file,"/root/arweave/apps/arweave/src/ar_http_iface_middleware.erl"},{line,1332}]},{ar_http_iface_middleware,handle,3,[{file,"/root/arweave/app...

[ldmberman]

Could make these requests to your node?

/chunk/5813282375
/chunk/5813020231
/chunk/5812758087
/chunk/5812495943
/chunk/5812233799
/chunk/5811971655
/chunk/5811709511
/chunk/5811447367
/chunk/5811185223
/chunk/5810923079
/chunk/5810660935

/chunk/7967313818
/chunk/7967051674
/chunk/7966789530
/chunk/7966527386
/chunk/7966265242
/chunk/7966003098
/chunk/7965740954
/chunk/7965478810
/chunk/7965216666

/chunk/5216485801
/chunk/5216223657
/chunk/5215961513
/chunk/5215699369
/chunk/5215437225
/chunk/5215175081
/chunk/5214912937
/chunk/5214650793
/chunk/5214388649
/chunk/5214126505
/chunk/5213864361
/chunk/5213602217

It will force it to validate the index for these chunks (will return 404 if invalid), remove the invalid onces, and subsequently re-sync them.

[ThomasBlock]

Thank you for the feedback! They all worked for me and nothing changed..

/chunk/5213602217
{"tx_path":"XogBub87VLLDgYZpqUR_2ITm1W-qnyCJeEE1vTGQqJC0JKwjqAzEFr37lcUqYJXnn1YYcglHL88DdaJ8Uhw4HgAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAkvYL7R-PUf57CjABcjC-2ZclL3AYAOjh0njhAdjgixNv8kAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAzQRA","packing":"unpacked","data_path":"l2jSZoeFc5JwuyQZ7nmBZFMxlVdG_CGFfUGR4Y7ik1LliElKYjZjzFGi5_JiTGVE7KSUdUYXNHtt9_sQzdmElgAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACAAAxoqDxVecPnipFa8yHrQYEEaF2scgKd2qU39iNHaax8wAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAqEbA","chunk": ...

[ldmberman]

Ah, my bad, I forgot about the block y0vlt5dHDXB22pf_nqlHlDJfZcAhwadeuFOHTdl_XDeloQTj7h7mGuj76eRCGRvD:

/chunk/9307871445
/chunk/9307609301
/chunk/9307347157
/chunk/9307085013
/chunk/9306822869
/chunk/9306560725
/chunk/9306298581
/chunk/9306036437
/chunk/9305774293
/chunk/9305512149
/chunk/9305250005

Which errors do you still see?

[ThomasBlock]

Ah so maybe this is not related to the problem that you think of. no problem with your mentioned chunks.

it is not just three blocks. i see a different block every second..

2022-08-19T23:24:53.423260+02:00 [warning] /root/arweave/apps/arweave/src/ar_header_sync.erl:525 event: ar_header_sync_block_tx_root_mismatch, block: <<"Za4a_kw44RnuR0VEuyYPxtGGvY8Q3Y6d4V0aTF71Ncxunh9KLjbS_qe0I8yhVOrI">>
2022-08-19T23:24:55.504777+02:00 [warning] /root/arweave/apps/arweave/src/ar_header_sync.erl:525 event: ar_header_sync_block_tx_root_mismatch, block: <<"Su4Bq54ukesFJFsEREJooDYRLJUuV6yJlpF3CKM4XYcgYKrP7-HZNVuLpn4AJ9-a">>
2022-08-19T23:24:56.929489+02:00 [warning] /root/arweave/apps/arweave/src/ar_header_sync.erl:525 event: ar_header_sync_block_tx_root_mismatch, block: <<"b3UTJPRxtIbo06dcw0Q0e1WUSqf6fMaotQWK6b2Vfa-6ft9tp3Mq7l-mL3xrHsyI">>
2022-08-19T23:24:57.392364+02:00 [warning] /root/arweave/apps/arweave/src/ar_header_sync.erl:525 event: ar_header_sync_block_tx_root_mismatch, block: <<"q2BZKKKkeIsq2RnNHd5-Z2A6qEctlxPWb3ZZ0cRj9wCdkPHeJ_hcM_OnOAyo9qRu">>
2022-08-19T23:24:59.341112+02:00 [warning] /root/arweave/apps/arweave/src/ar_header_sync.erl:525 event: ar_header_sync_block_tx_root_mismatch, block: <<"Jurx1c_ch5tpihbtXjtsXEuOT9ywK9Nl2PkffmHV_A1LliBxjVYk9Da-mSR80yx6">>
2022-08-19T23:24:59.906473+02:00 [warning] /root/arweave/apps/arweave/src/ar_header_sync.erl:525 event: ar_header_sync_block_tx_root_mismatch, block: <<"HdAd99TXKNeeiUSPgXcKJeE9ptCE82uHeb7dTxhyvaWMl2V1gBs5nQqZoJ-1jCWm">>
2022-08-19T23:25:01.210502+02:00 [warning] /root/arweave/apps/arweave/src/ar_header_sync.erl:525 event: ar_header_sync_block_tx_root_mismatch, block: <<"V8PYiU7x1sLlQYncS6C5CBohz8nueNHxtS70sAsfbF061ufVfG3f0qKMHKgn-uJs">>
2022-08-19T23:25:02.023499+02:00 [warning] /root/arweave/apps/arweave/src/ar_header_sync.erl:525 event: ar_header_sync_block_tx_root_mismatch, block: <<"Uxtog8ANIRhC2ro8Vxo2vrWUFanieO7stzXUjCOfg8cqww5Eq2ghjWiIIoeSc9oY">>
2022-08-19T23:25:03.564121+02:00 [warning] /root/arweave/apps/arweave/src/ar_header_sync.erl:525 event: ar_header_sync_block_tx_root_mismatch, block: <<"NiTj_RA1UNLmTOI3uHk34vIVbOAU4AQZAJbSPY-8H5Fu3wPTmcPI6TCMQMVwGE_1">>
2022-08-19T23:25:03.865670+02:00 [warning] /root/arweave/apps/arweave/src/ar_header_sync.erl:525 event: ar_header_sync_block_tx_root_mismatch, block: <<"ug2Abyx6lDC1T58aYFyFgTTV8qMe8fYcSYHItGfqGVyWzhGY1zvfYJtv5X0oH9aD">>
2022-08-19T23:25:05.652032+02:00 [warning] /root/arweave/apps/arweave/src/ar_header_sync.erl:525 event: ar_header_sync_block_tx_root_mismatch, block: <<"s9_yul8ej7ScWzLogKP6btgmNhnN54u6z3dePiCDFVRqkUqJ-rP_HrVV9GxrM942">>
2022-08-19T23:25:05.829342+02:00 [warning] /root/arweave/apps/arweave/src/ar_header_sync.erl:525 event: ar_header_sync_block_tx_root_mismatch, block: <<"CyosTFY9IFSSMMFmxYKdCOESHTuNq2K1dTQb58PIxX_7RLpSSKlarfHA9vzDcmlR">>
2022-08-19T23:25:08.089316+02:00 [warning] /root/arweave/apps/arweave/src/ar_header_sync.erl:525 event: ar_header_sync_block_tx_root_mismatch, block: <<"-Xj8i7M4eF49V2F9IzJ3qTinY0ZcPzazJ8JvuBNbFS6veU2uq04RUlufZxZyF4Sx">>
2022-08-19T23:25:08.457074+02:00 [warning] /root/arweave/apps/arweave/src/ar_header_sync.erl:525 event: ar_header_sync_block_tx_root_mismatch, block: <<"YsAxakZK621xI8-K72mj4N9bYy6gA6FhXT-hywnaELHBMHG4FlaMOmP3xw9Lgt03">>
2022-08-19T23:25:10.442429+02:00 [warning] /root/arweave/apps/arweave/src/ar_header_sync.erl:525 event: ar_header_sync_block_tx_root_mismatch, block: <<"1wDK_AMvUUfTQtCRuwK2VbdgfACMkEkkzJ7rd_GblBYo8qP6M5IJ9MSofMCuhrdY">>
2022-08-19T23:25:10.920668+02:00 [warning] /root/arweave/apps/arweave/src/ar_header_sync.erl:525 event: ar_header_sync_block_tx_root_mismatch, block: <<"Z6j-eiWDN8ZsfYCHO5UszlcIMJVaGveuhW_SZcjMrKIIxxNQa-V8eAMPHR13e3pq">>
2022-08-19T23:25:12.516895+02:00 [warning] /root/arweave/apps/arweave/src/ar_header_sync.erl:525 event: ar_header_sync_block_tx_root_mismatch, block: <<"FCvc52ZbXJcfl_oTNUZgYvw7ac2M8e_2WSLQN7O12j9-dAFSKcU5SSFqsO8yoz83">>
2022-08-19T23:25:13.287576+02:00 [warning] /root/arweave/apps/arweave/src/ar_header_sync.erl:525 event: ar_header_sync_block_tx_root_mismatch, block: <<"ShH_EMhwAUhivJkVphSwplQjlEKzSVjkByqq93STyoV0MoZD5ioVtB_uE-LCxjCa">>
2022-08-19T23:25:15.379771+02:00 [warning] /root/arweave/apps/arweave/src/ar_header_sync.erl:525 event: ar_header_sync_block_tx_root_mismatch, block: <<"ui21Nkctec3vaR-rYn8XSGYGHZ4DS_Bb3V-VS6vHWO5Sth2ifLbJ_-h1frNPdJ7S">>
2022-08-19T23:25:16.349855+02:00 [warning] /root/arweave/apps/arweave/src/ar_header_sync.erl:525 event: ar_header_sync_block_tx_root_mismatch, block: <<"erB92nN3W5b8J8MEt80z40dQ-1A1oVAU4dfVYprFissQJap1p7V5v2KYs0Zzrmf_">>
2022-08-19T23:25:17.727637+02:00 [warning] /root/arweave/apps/arweave/src/ar_header_sync.erl:525 event: ar_header_sync_block_tx_root_mismatch, block: <<"VwzatUzVswnX5ir5OtVI0sHil3bEtcnRSqHu2DyRXOYLI0l_eEORdz6bEMCx1c7v">>
2022-08-19T23:25:18.039423+02:00 [warning] /root/arweave/apps/arweave/src/ar_header_sync.erl:525 event: ar_header_sync_block_tx_root_mismatch, block: <<"RkIExxrMEY-ufpcYRlTPmebpm-oR76PWDfbo2w2-LVbC9iyCWsYyyg7OI8E5mWZ4">>
2022-08-19T23:25:20.446634+02:00 [warning] /root/arweave/apps/arweave/src/ar_header_sync.erl:525 event: ar_header_sync_block_tx_root_mismatch, block: <<"DOJuSKgfxTPVwSup3D0m8ArkSW5Dh62yxzswjkaKfr1pGZgdfsEomLEB0z3YaB1P">>
2022-08-19T23:25:21.263818+02:00 [warning] /root/arweave/apps/arweave/src/ar_header_sync.erl:525 event: ar_header_sync_block_tx_root_mismatch, block: <<"MeZkiiD4DZwf6uf7M7O23ldDtUzFfAg01TbGzP42ASZaSnNvFeEJ-8wQs16ud1tI">>
2022-08-19T23:25:23.806321+02:00 [warning] /root/arweave/apps/arweave/src/ar_header_sync.erl:525 event: ar_header_sync_block_tx_root_mismatch, block: <<"8if-00_tLzMdpNpHSnv40c_h7s50DEMCrJOfDNvHBUMG4AawhCPjVU7hrze0erEs">>
2022-08-19T23:25:24.931563+02:00 [warning] /root/arweave/apps/arweave/src/ar_header_sync.erl:525 event: ar_header_sync_block_tx_root_mismatch, block: <<"A9xCbxl4CTJ7eZKAo6brwZ3Rtb22U73Q1RN_5AYUvkyRQLAWWKyH_NwA9EBY9Idk">>
2022-08-19T23:25:26.940792+02:00 [warning] /root/arweave/apps/arweave/src/ar_header_sync.erl:525 event: ar_header_sync_block_tx_root_mismatch, block: <<"okf7GdiRW8mV0oC8Lqwzn7E48r9QAPsTv8SMpz-sfFx1B-lNFHuMzF09l24Zv6le">>
2022-08-19T23:25:30.305564+02:00 [warning] /root/arweave/apps/arweave/src/ar_header_sync.erl:525 event: ar_header_sync_block_tx_root_mismatch, block: <<"NCspRi7XxYIVNW7iFbTygxaMLh1fudI2bk0ZgikT3ubgjtYn8FEu_BESR6-h322p">>
2022-08-19T23:25:32.073495+02:00 [warning] /root/arweave/apps/arweave/src/ar_header_sync.erl:525 event: ar_header_sync_block_tx_root_mismatch, block: <<"xmvt8Dppju1I3XMmbimm8H9OANllvDMGI7XP20zRe2TeW4sW7JsFBvdvjA0E99IF">>

[ThomasBlock]

Update on this: The error was fixed after restoring a backup ( no old backup, just one day back ). In Principle not much has changed. But maybe there was a problem with restoring some files, as i see in the last twelve hours thousands of the following messages - and i see that the value of "v2_index_data_size" is slightly shrinking.

2022-08-28T21:21:26.375868+02:00 [warning] /root/arweave/apps/arweave/src/ar_data_sync.erl:1135 event: invalidating_bad_data_record, type: 2, range_start: 72581978562806, range_end: 72581978824950

But the good news is that the missing blocks could be downloaded. no more ar_header_sync_block_tx_root_mismatch, i could reach 100% blocks.

[ldmberman]

Interesting, thank you for the update @ThomasBlock!