Chain inconsistency from testnet Heartwood activation

[str4d]

I ran my node that was pre-Heartwood, and it synced up through Heartwood successfully. Restarted to add mineraddress to my config, and hit the following error:

2020-05-05 00:41:58 ERROR: LoadBlockIndex(): block index inconsistency detected (hashLightClientRoot != hashChainHistoryRoot): CBlockIndex(pprev=0x55f4e6c1d600, nHeight=903800, merkle=d2277b2d66b2faa3f6ac6cd2fa6f0655130f4a7a19cab97076ce5024ee203e9b, hashBlock=0044f3c696a242220ed608c70beba5aa804f1cfb8a20e32186727d1bec5dfa1e)

I think this is the same issue that others are seeing. Not sure yet whether it is resolved by a reindex, or whether the issue persists.

[str4d]

The comparison failure is:

ERROR: LoadBlockIndex(): block index inconsistency detected (hashLightClientRoot 626444395cd5963d3dba2652ee5bd73ef57555cca4d9f0d52e887a3bf44488e2 != hashChainHistoryRoot 0000000000000000000000000000000000000000000000000000000000000000): CBlockIndex(pprev=0x562cce3a4630, nHeight=903800, merkle=d2277b2d66b2faa3f6ac6cd2fa6f0655130f4a7a19cab97076ce5024ee203e9b, hashBlock=0044f3c696a242220ed608c70beba5aa804f1cfb8a20e32186727d1bec5dfa1e)

hashLightClientRoot is meant to be set to null (all-zeroes) for the Heartwood activation block, but that is not what is being recovered from the chain index. Maybe a write-read round-trip inconsistency?

[str4d]

$ ./zcash-cli getblock 903800
{
  "hash": "05688d8a0e9ff7c04f6f05e6d695dc5ab43b9c4803342d77ae360b2b27d2468e",
  "confirmations": 227,
  "size": 1630,
  "height": 903800,
  "version": 4,
  "merkleroot": "d96ba8c0e9aa658904195206a36c075e5306a0fa347f9712fb1a6249bd770b0d",
  "finalsaplingroot": "626444395cd5963d3dba2652ee5bd73ef57555cca4d9f0d52e887a3bf44488e2",
  "chainhistoryroot": "0000000000000000000000000000000000000000000000000000000000000000",
  "tx": [
    "d96ba8c0e9aa658904195206a36c075e5306a0fa347f9712fb1a6249bd770b0d"
  ],
  "time": 1588624149,
  "nonce": "00006b7b3958a3dfd519c9f7b5b2e996e6cde1a3f29303001dd5fe622424000c",
  "solution": "006444054102c3ee24f8b3ef5ffedd31f84833082405348abf3a0df56799fa7712d04bd4b6c4eebaaef20b238bc2d291044701f51400d57805159a6b8cff9462a7bcc11ba62a0fb4bd167a3d54595f03205e99d801fd974ada469bc679c4087e9873b9f69470d6f9500c9ebe35df49b5c1846e0176e0c0944598a6dfa4830513c08a4c81b60aec7ffae13277d64311175a341e1282fc35889cd48ffa4a958026430f39c82171759f08d9e4a1c166fb5973ea1b5a02713ac348b75ef4271cc8d6535a55a0dedce24221d7e19cf5522a116a6c0a56333b4bca439ad6b434c85a41c755ad317322c70f3b70d93344abe12c20c232644f87ccb945773c870dd8afd71511bf73e7cd679942e4bbcfab38fe6c881c9899e7bb17176d1b0892fa43ffc032f6fb7e587c137538d018626277ed3b46b07963abe2a6895664e3654effc4a85d295f663c98e8aa69f0f2775a38384900dc9ae867dc6a6923b741d92be90fbcc9fd99be0f38218a33382eb891834c467c12f077e9a770cdfe4205ca251f524ba930c97714859248be35ec6b77065516d77f78cdc7e639f196a88d2d765a4eaf5cf624e00b534e77dd14e129ec8f32aecc2fec12e671fddbca1140e4020084ec8343a995ae3a2dba9ea3c03a27a31d81bf08088c6fecfd622208baea1f122db67db6fc1ec9adff8cdb861ce61c12742f20689ad7725f8baf026f30c5a651b31716fec25f79c38673b21f5e27e6074c174e0b0fcae3a0f1d241b54735a0e09655c2170ab680dafb5707039c23730979ef1e1d23808feb5c321186726a9557c337bc95b3576f59a985b83eb0bf0825f043a25731055c40c407aba3784599f1f5f9cb0f59f1723f5cd251ce045244334c3f9518a50c4bf10d4de44fdc9b658bdea4796dce75e27685df9690a24a2c1b231514dd9decac95a3b0d5a3ea9e085a4b1b009f5182d000f24f8bd4d401a67f10c625eb5736360ab3e4aa731a965d13c8d19b3bec4a08bf696c5b4a08055bdc9358d0d904c7f5e6f1487ba675ddf4a69e2327cefe20a1bdf1f6f46326e92aa580d43e9f560909f4e480f3cd93b110ce6648e639a706c1281fc5a2123c7a3ebc9694fd4fd2a1613a9dd2020e33bf252d2b247e7761e0ac1d58cde38e61abd92928d6d2d7c82e689284381e284948803693b0b61c76a53d38a000016458c011dbc379047950dc535fb14d37acfe5ed9238d76bcf19af820df9884d0a8f89ebd6e563484150613d5b852d927a77f68822b25ffdfcdd04d950e0b1efdae45f4764dbbff801222f966851b097799a899026f2c500062fc953f7fa5c4de5fa87db5ac1edad123abb152838d0616b2fcf79ec85a2ffe444b55128f03dcc4f730238be5a89870890a6a6fa0efd18bc02c2a28d5604f0f6a40845183f26e3c495a48583cf6a400f99ecbc6cc9212a544325962241f96661bd5f5ee039aa4132582aba86372a8789868c1c231e55b2a20047caa1116940a2faefd330584a33388da499a26181a9645b10af7ff51d78441d912a14a52fc8b9c9acd03b0e0b49256a7d3f6b046ac2bf757a1bf789cb71e436ea7e222f3ebc3b354847af459047931d252bb3412e5eaa12daea9f18ce1f2a3167faa69dab4b4884c3288955eac5d81310a9e55fd6c549155d592b8bbb7081ef26331e74911faa7e3d9c5a23352ded81dd4df1eb7f1a6a85c54b5f86d631d2ae91c056fa558718c1200638804dd4a0fa4825a21da6f04deddaa77a8dc20eba3bec6a09e673b36a443baaa27316ca88f1a9c09be536c6956df9778f0d394dcc43ab26d7259a087141fd25dcb6402a7845b140b2b2514c9e948ff23aa23205fde39daf1ff75301295bf424595a615551af625ac3de0e34fa6f4abddd8fb1475e6ce8bf79b2d29",
  "bits": "2007ffff",
  "difficulty": 1,
  "chainwork": "00000000000000000000000000000000000000000000000000000025ba29b8d3",
  "anchor": "fc66f64cc4f03d8811e6a4b8d2b4ca893c32017b7914e7b77d15c07ea6ff3f23",
  "valuePools": [
    {
      "id": "sprout",
      "monitored": true,
      "chainValue": 440632.99730484,
      "chainValueZat": 44063299730484,
      "valueDelta": 0.00000000,
      "valueDeltaZat": 0
    },
    {
      "id": "sapling",
      "monitored": true,
      "chainValue": 106271.32545645,
      "chainValueZat": 10627132545645,
      "valueDelta": 0.00000000,
      "valueDeltaZat": 0
    }
  ],
  "previousblockhash": "001296b0b05b3818f58d87bc6b24fe7a247771c9fd59c656a49163e94d7ec1fb",
  "nextblockhash": "04e6fc4acc588f2df706b252c9d1f7f034d1fa829299adc2ba0a923bd648bc06"
}

So it looks like hashLightClientRoot is being set to hashFinalSaplingRoot for the Heartwood activation block, instead of to null.

[str4d]

Ah, this is caused by there being testnet miners that hadn't upgraded. The chain inconsistency check is triggered by a block header's height, but zcashd is storing now-invalid block headers for the left-behind chain:

• The testnet Heartwood activation block is 05688d8a0e9ff7c04f6f05e6d695dc5ab43b9c4803342d77ae360b2b27d2468e.
• The equivalent-height block for the non-Heartwood chain is 0044f3c696a242220ed608c70beba5aa804f1cfb8a20e32186727d1bec5dfa1e.

They both happen to have the same hashFinalSaplingRoot because neither block included any Sapling transactions.

The solution will be to only enforce that Heartwood consistency check for block headers that are marked as valid in the chain DB (as we can't make any assertions about invalid block headers).

[str4d]

Closed by #4499.

[daira]

#4499 was an insufficient fix, because it did not consider the case where a post-Heartwood node wrote a block index object for a header from a non-upgraded peer. In that case the version in the block index entry is >= CHAIN_HISTORY_ROOT_VERSION, and so the fix in #4499 has no effect. In addition, we should skip the consistency check when the index object validity is not BLOCK_VALID_CONSENSUS.

[daira]

The remaining issue was fixed by #4503.