Overwinter: after a block rewind, syncing to the correct branch can fail, and reindex is required

[bitcartel]

We found an issue in a recent test of activating Overwinter on Testnet, specifically with block rewinding.

Background:

Testnet

• Sprout nodes (running 1.0.15)
• Overwinter nodes (running custom 1.0.15 called 'Federer' set to activate at block 200000 instead of the scheduled 207500)

Test 1 (Success)
Both branches of the fork mine blocks successfully. We stop a Sprout
node and launch a Federer node using the datadir of the Sprout node. As
expected we see a message asking us to perform a reindex since the
Sprout chain is >= 100 blocks ahead of the fork point.

Test 2 (Failed)
Both branches of the fork mine blocks successfully but we stop the
Federer mining early (it was around 25 blocks). We stop and launch a
Sprout node using the datadir of the Federer node. It rewinds to block
199999 but cannot sync with the Sprout chain, because block 200000 was
previously invalidated by the Federer node and assigned the state:
BLOCK_FAILED_VALID | BLOCK_VALID_TREE.

We believe this issue is because we have not changed the block header
format from Sprout and the Overwinter network upgrade. So peers
advertising the same protocol version stay connected and send block
headers to each other which validate fine, but the blocks are not valid
as they use different transaction formats.

See here for more info:

https://chat.zcashcommunity.com/channel/zcash-dev?msg=29QaGdz9nSFZsmDQy

https://chat.zcashcommunity.com/channel/zcash-dev?msg=z4ph6M5GgJkdea4FC

[str4d]

This is now closed by #3132.