detect and resolve chain reorganization

[borngraced]

Verify the parent hash of each block to confirm its consistency with the hash of the preceding block. If it does not match, proceed to backtrack until a match is found. Then, proceed with indexing from the block height of the matched block.

fixes task 7 in #1612

[shamardy]

@borngraced can you please fix CI errors? please also merge with latest dev for CI changes :)

[borngraced]

ready now @shamardy

[shamardy]

@borngraced can you please check my latest changes? You are the reviewer now 😅

[borngraced]

First review check without testing the code
I'll do another review after testing the code

[onur-ozkan]

Thanks for the enhancement!

Few notes from my side:

[onur-ozkan]

LGTM once CI is green(except the common fails)

[shamardy]

@borngraced this PR is ready for next review from you.

[borngraced]

I try testing the code to see if headers will be downloaded and stored as supposed but unfortunately they're not. I was getting Permanent error: Unknown destination address on my console.

Permanent Error: rpc_clients:2113] JsonRpcError { client_info: "coin: RICK", request: 
JsonRpcRequest { jsonrpc: "2.0", id: "34", method: "blockchain.block.headers", params: 
[Number(2), Number(16)] }, error: Internal("Unknown destination address 
electrum3.cipig.net:10017") }, on retrieving headers from server electrum3.cipig.net:10017

[shamardy]

I try testing the code to see if headers will be downloaded and stored as supposed but unfortunately they're not. I was getting Permanent error: Unknown destination address on my console.

Permanent Error: rpc_clients:2113] JsonRpcError { client_info: "coin: RICK", request: 
JsonRpcRequest { jsonrpc: "2.0", id: "34", method: "blockchain.block.headers", params: 
[Number(2), Number(16)] }, error: Internal("Unknown destination address 
electrum3.cipig.net:10017") }, on retrieving headers from server electrum3.cipig.net:10017

@borngraced I guess this was during coin activation not after since it tries to get blocks 2-16, you should have checked task::enable_utxo::status to know that the coin wasn't activated and returned this error which is the desired approach, since if there was a problem with one of the electrum servers on activation, this error indicates that the user shouldn't use this electrum server. Please remove electrum3.cipig.net:10017 from the electrum list and try again, please also provide me with the logs that led to that problem without removing electrum3.cipig.net:10017 to make sure that the problem was from the server, in the logs there should be a log that indicates why the url was removed from the connections list.

[cipig]

RICK electrums are fine

but keep in mind that every electrum server is restarted once a week, so you will loose connection if you do anything during the restart

[shamardy]

RICK electrums are fine
but keep in mind that every electrum server is restarted once a week, so you will loose connection if you do anything during the restart

The problem is if the connection was lost during activation only, if it was while mm2 is running, mm2 shouldn't remove the server and will use the other servers until this server is back. I also remember that you mentioned this restart to me while working on an old issue more than a year ago, I remember it was in tuesday if I am not mistaken, so maybe that's what happened when @borngraced was testing.

[cipig]

I also remember that you mentioned this restart to me while working on an old issue more than a year ago, I remember it was in tuesday if I am not mistaken

They are restarted at different times

50 9 * * 2 electrum1
50 17 * * 4 electrum2
50 1 * * 6 electrum3

the first 2 numbers are minute and hour... the last number is the weekday... 2, 4 and 6... "day of week 0–7 (0 or 7 is Sun, or use names)"... so electrum1 is Tuesday

[borngraced]

I try testing the code to see if headers will be downloaded and stored as supposed but unfortunately they're not. I was getting Permanent error: Unknown destination address on my console.

Permanent Error: rpc_clients:2113] JsonRpcError { client_info: "coin: RICK", request: 
JsonRpcRequest { jsonrpc: "2.0", id: "34", method: "blockchain.block.headers", params: 
[Number(2), Number(16)] }, error: Internal("Unknown destination address 
electrum3.cipig.net:10017") }, on retrieving headers from server electrum3.cipig.net:10017

@borngraced I guess this was during coin activation not after since it tries to get blocks 2-16, you should have checked task::enable_utxo::status to know that the coin wasn't activated and returned this error which is the desired approach, since if there was a problem with one of the electrum servers on activation, this error indicates that the user shouldn't use this electrum server. Please remove electrum3.cipig.net:10017 from the electrum list and try again, please also provide me with the logs that led to that problem without removing electrum3.cipig.net:10017 to make sure that the problem was from the server, in the logs there should be a log that indicates why the url was removed from the connections list.

Weird, but it's working fine now. tested with BTC also!

[shamardy]

Weird, but it's working fine now. tested with BTC also!

There is an ignored test that tests the BTC block headers sync that I always test before pushing :), that's how I knew this was working
https://github.com/KomodoPlatform/atomicDEX-API/blob/ef82cb7b481fb9ddf60d0c81027f3fc7db0a5322/mm2src/mm2_main/tests/mm2_tests/mm2_tests_inner.rs#L7257-L7261

[borngraced]

LGTM 👍🏼 well-done