ChainValidationUpdateError for cardano-node 1.10.0 on staging

[ArturWieczorek]

What: cardano-node 1.10.0
Environment: staging

My node just stopped sync at 44th epoch with following errors (Those are 2 last log messages, the process in hanging):

[artur-de:cardano.node.ChainDB:Error:37] [2020-04-10 08:37:54.09 UTC] {"kind":"TraceAddBlockEvent.AddBlockValidation.InvalidBlock","error":"ExtValidationErrorLedger (ChainValidationUpdateError (SlotNumber {unSlotNumber = 969188}) (Registration NullUpdateProposal))","block":{"kind":"Point","slot":969188}}
[artur-de:cardano.node.ChainDB:Error:37] [2020-04-10 08:37:54.09 UTC] {"kind":"TraceAddBlockEvent.AddBlockValidation.InvalidCandidate","block":"aaffe2c@969188"}

mainnet and testnet fully synced without any errors so this happens only on stage.

[ArturWieczorek]

It stopped working with this commit: cf2d053

Based on error message it is pointing most likely to:
IntersectMBO/cardano-ledger@90b14c0

CC: @nc6

[disassembler]

For context, a number of update proposals happened in close proximity here:
csl-daedalus:9

{
  "body": {
    "updatePayload": {
      "votes": [
        {
          "signature": "6b5ac17c338334405e546a52cc2a37f8385801e793e29a578fa3563197fb24b8be72e91f4eafe4c8595222541e3e96afde597eba305ce93ed9d40055675fbf0a",
          "voterVK": "sc4pa1pAriXO7IzMpByKo4cG90HCFD465Iad284uDYz06dHCqBwMHRukReQ90+TA/vQpj4L1YNaLHI7DS0Z2Vg==",
          "proposalId": "c50c43724c639779ac35670a09ed209c4f0d70a5e330e8a5ab23fbfb6be68f51"
        },
        {
          "signature": "00155135afba326c8377b4fc412ea6ece341167a01f50cb7194790ee0c4023a704c62719a4402b6264897074b24881fcc3c1e897d85897cff077d70925185f08",
          "voterVK": "FVT9+EybM5YFJKmYZoUa0hb6mZEZXmtChhxwpTS/Gn8QeIaDzpFBq889IXTCt5Idew+JT+9f+tiXAluAlCNtng==",
          "proposalId": "c50c43724c639779ac35670a09ed209c4f0d70a5e330e8a5ab23fbfb6be68f51"
        },
        {
          "signature": "d742d6f6f3e7515b51de71c61e7fdc2f322c253067a4d6748649dbf865d45c05d6cefedcd2825c3dd2a1f366171efefe1aa4012fc0ec42727d6c692bd8435604",
          "voterVK": "HV5+gKgoYbGoAB34NMkIDvKBrulsRyfCZmS0Fizrl7bUnEfMs9/EyvLcgDaqLdfNLNfFzBSgAR1QWNWADQ0Gig==",
          "proposalId": "c50c43724c639779ac35670a09ed209c4f0d70a5e330e8a5ab23fbfb6be68f51"
        },
        {
          "signature": "4d91c89d5c24f8b87c3b010836db2f4191d9a79d4d294b5b6a0bf3e47f80fe699d552ba58e3cbac0cb8e28ac75fde9c76615865aae03ab3272f203c8e150be08",
          "voterVK": "9nyylndB1x0liSdnz0LhpoufvOpEuZ1jJLcrDZYspFk/mj10EkWZUCi/wfiST0hbeunKp2lhx+HbGpNuMlEvEA==",
          "proposalId": "c50c43724c639779ac35670a09ed209c4f0d70a5e330e8a5ab23fbfb6be68f51"
        }
      ],
      "proposal": {
        "signature": "0e6530be971270d6a0e5093cc1cb76040b0d3ea3b740bc36d632959a9067b87538fe16c31b58a8d3084381c5c4b1765733c283d480eeeacd6fbc69d7bcc66807",
        "body": {
          "protocolVersion": {
            "minor": 1,
            "major": 0,
            "patch": 0
          },
          "parametersUpdate": {
            "txFeePolicy": null,
            "maxTxSize": null,
            "scriptVersion": null,
            "updateProposalTheshold": null,
            "maxBlockSize": null,
            "maxHeaderSize": null,
            "mpcThreshold": null,
            "slotDuration": null,
            "maxProposalSize": null,
            "softforkRule": null,
            "heavyDlgThreshold": null,
            "updateVoteThreshold": null,
            "unlockStakeEpoch": null,
            "updateProposalTTL": null
          },
          "metadata": {
            "win64": "2690a1cca49158ecfd18cc1969e8c29632c8115543cb44859c62ccc23f1c5830",
            "macos64": "dd89d4297159cfc63dc3f0ac4b49e20c2a06ef8a76c3e5c2cdaf050b9197864a"
          },
          "softwareVersion": {
            "appName": "csl-daedalus",
            "number": 9
          }
        },
        "issuer": "sc4pa1pAriXO7IzMpByKo4cG90HCFD465Iad284uDYz06dHCqBwMHRukReQ90+TA/vQpj4L1YNaLHI7DS0Z2Vg=="
      }
    },
    "txPayload": [],
    "dlgPayload": []
  },
  "header": {
    "signature": {
      "signature": "85f96af3cb2c95ac9c1f9faaff37e8c9aa9ac3002ea8ce480c782ac4f94aaae7da1ca92a338e54dc472b14e1adea4d35babe28c72fc1db39119cfa935dfcac0b",
      "dlgCertificate": {
        "signature": "b727b58fb68157b11dbcc67243246e87237bbb1ab14513033ce329565a8c31bae2fb3dcd43b2a83fe22800fc9429414e79a1c81c5744c27c21e729c6d367a70f",
        "delegateVK": "1C4vGdgHil59gukd8YPsqtV2pgfalL4j7phWuKaEisLkCedCusGtaskOwk0fkiEgU8kVc1TFKjqxTEQYdE3DpA==",
        "issuerVK": "kJbDly+8W5I9AFWLruJ9QL5BbJvakhUR4GqhtBw4h28m04+r0b+tqySnJLeMAHbX9PUc52j6QZ4XxEAuUTyStw==",
        "epoch": 0
      }
    },
    "proof": {
      "utxo": {
        "root": "0e5751c026e543b2e8ab2eb06099daa1d1e5df47778f7787faab45cdf12fe3a8",
        "witnessesHash": "afc0da64183bf2664f3d4eec7238d524ba607faeeab24fc100eb861dba69971b",
        "number": 0
      },
      "delegation": "afc0da64183bf2664f3d4eec7238d524ba607faeeab24fc100eb861dba69971b",
      "update": "4d2b66ae3b30389ba68aea30c82dd6e5a592888fc8bcd5bace544f48b27594ba"
    },
    "protocolVersion": {
      "minor": 1,
      "major": 0,
      "patch": 0
    },
    "genesisKey": "kJbDly+8W5I9AFWLruJ9QL5BbJvakhUR4GqhtBw4h28m04+r0b+tqySnJLeMAHbX9PUc52j6QZ4XxEAuUTyStw==",
    "protocolMagicId": 633343913,
    "slot": 965869,
    "softwareVersion": {
      "appName": "cardano-sl",
      "number": 0
    },
    "prevHash": "351d547575e69b077dc0bff1eaee4061839e34e284a49632c57a2e9a4210a922",
    "blockHeight": 963485
  }
}

csl-daedalus:10 (accepted)

{
  "body": {
    "updatePayload": {
      "votes": [
        {
          "signature": "b24ca29af8931db939f53918e74aa0bb64b97faf4ad45804599fe1a485d9c7981395f94f5ab0209ab11e25f3ca585180c0b56e57e3ffc861f940f9bdd0d05c0d",
          "voterVK": "sc4pa1pAriXO7IzMpByKo4cG90HCFD465Iad284uDYz06dHCqBwMHRukReQ90+TA/vQpj4L1YNaLHI7DS0Z2Vg==",
          "proposalId": "5766631341e84f13b8c683d8d4b3745fc79150a7c6ebbca9c381c670b2fa7f35"
        },
        {
          "signature": "3135d41a2b2d1e2979eca933501158415d7dd787240e9302fb67074efb187e780f5e8fcac32378367fb70a0433c82c445d72dbddddd1cd397f1b5c9603979c07",
          "voterVK": "FVT9+EybM5YFJKmYZoUa0hb6mZEZXmtChhxwpTS/Gn8QeIaDzpFBq889IXTCt5Idew+JT+9f+tiXAluAlCNtng==",
          "proposalId": "5766631341e84f13b8c683d8d4b3745fc79150a7c6ebbca9c381c670b2fa7f35"
        },
        {
          "signature": "43c0b7ac8de9924bcdcb824cc890de347f0bb008d138dba682fa21fba788aa0833ef44bb08d6b2491c674b10021cb961c8dd8c3b07a0fe30c3ff19a93cb87f04",
          "voterVK": "HV5+gKgoYbGoAB34NMkIDvKBrulsRyfCZmS0Fizrl7bUnEfMs9/EyvLcgDaqLdfNLNfFzBSgAR1QWNWADQ0Gig==",
          "proposalId": "5766631341e84f13b8c683d8d4b3745fc79150a7c6ebbca9c381c670b2fa7f35"
        },
        {
          "signature": "244d8e0b63aa89e1520976357995d44e7f9084d8f76ee79bff4aa487311ae838a09a6a295a9d7117159e5e31f8b218af18108c9afde3f779c3af21c5dbea9307",
          "voterVK": "9nyylndB1x0liSdnz0LhpoufvOpEuZ1jJLcrDZYspFk/mj10EkWZUCi/wfiST0hbeunKp2lhx+HbGpNuMlEvEA==",
          "proposalId": "5766631341e84f13b8c683d8d4b3745fc79150a7c6ebbca9c381c670b2fa7f35"
        }
      ],
      "proposal": {
        "signature": "e0b31f63ca75a93b107c6fcd7410ce9a0145e4ff8c6b22790cf73e43e0f54de6f6f418cf9a996b1f01b6f9914016d061abd0b5d362134396d28e675d2f5a7f0a",
        "body": {
          "protocolVersion": {
            "minor": 1,
            "major": 0,
            "patch": 0
          },
          "parametersUpdate": {
            "txFeePolicy": null,
            "maxTxSize": null,
            "scriptVersion": null,
            "updateProposalTheshold": null,
            "maxBlockSize": null,
            "maxHeaderSize": null,
            "mpcThreshold": null,
            "slotDuration": null,
            "maxProposalSize": null,
            "softforkRule": null,
            "heavyDlgThreshold": null,
            "updateVoteThreshold": null,
            "unlockStakeEpoch": null,
            "updateProposalTTL": null
          },
          "metadata": {
            "win64": "2212502699039a859989f1ba1b1d758b21651690c49dff048394936e207d5d62",
            "macos64": "21398c0a46654d56e78e6cbf84ab4f8c4599ce0eb11c7193ae2a2a71ac3399a8"
          },
          "softwareVersion": {
            "appName": "csl-daedalus",
            "number": 10
          }
        },
        "issuer": "HV5+gKgoYbGoAB34NMkIDvKBrulsRyfCZmS0Fizrl7bUnEfMs9/EyvLcgDaqLdfNLNfFzBSgAR1QWNWADQ0Gig=="
      }
    },
    "txPayload": [],
    "dlgPayload": []
  },
  "header": {
    "signature": {
      "signature": "0a4fc92e82df3777ea4290109cf9b40bf420a07af34fcf793386ce44ce7a297d6c01690d07a9d58fb825a692a7fb29ac026a3633eeafc80fc12d084bc2f0fb01",
      "dlgCertificate": {
        "signature": "05ff34e3b7e703fb222609004361f3e03d45231e80064c0e8ccd4d164ad2e399b2544223cfdfef9dec067bf8395e0a6fd82735abfb8827bfc9bbb4648c73f504",
        "delegateVK": "LxnxuG5pTlojgG7TCjSsrrQTEWIpd5Lpakd/Jlmh3J1N5vIPAlLn5W8WqnSTj94W+OhjYzD3QN5vWJqDwqA8Sw==",
        "issuerVK": "Xf8mIEqV4YadaLHas2Q2zCg6hSXDc02QXq7Qvq90hR5EIGYIkqGcShF8pQoUAUQMrd8W+udqC6Cr///FzNO6YA==",
        "epoch": 0
      }
    },
    "proof": {
      "utxo": {
        "root": "0e5751c026e543b2e8ab2eb06099daa1d1e5df47778f7787faab45cdf12fe3a8",
        "witnessesHash": "afc0da64183bf2664f3d4eec7238d524ba607faeeab24fc100eb861dba69971b",
        "number": 0
      },
      "delegation": "afc0da64183bf2664f3d4eec7238d524ba607faeeab24fc100eb861dba69971b",
      "update": "e888f3ab58bdca4c4d859bc85a42938bac7cad4e1093eb34461b57970db47f90"
    },
    "protocolVersion": {
      "minor": 1,
      "major": 0,
      "patch": 0
    },
    "genesisKey": "Xf8mIEqV4YadaLHas2Q2zCg6hSXDc02QXq7Qvq90hR5EIGYIkqGcShF8pQoUAUQMrd8W+udqC6Cr///FzNO6YA==",
    "protocolMagicId": 633343913,
    "slot": 969023,
    "softwareVersion": {
      "appName": "cardano-sl",
      "number": 0
    },
    "prevHash": "1f147daac3722e317a430c3fa2b53006a01accb8a21c3e4396163a258a830c15",
    "blockHeight": 966639
  }
}

csl-daedalus:10 (ignored and causing issue with new node)

{
  "body": {
    "updatePayload": {
      "votes": [
        {
          "signature": "56cdc0b6afba042dab85ddffd317387505cfebe7cbe625b69ca1f2e6bc0a8bd010b0d5f54cc06dcb24dba6f9f95f0257e0c7cbe8cf571ee00c9289738847bd09",
          "voterVK": "sc4pa1pAriXO7IzMpByKo4cG90HCFD465Iad284uDYz06dHCqBwMHRukReQ90+TA/vQpj4L1YNaLHI7DS0Z2Vg==",
          "proposalId": "530633b209d8c29c65306e5c6b615f0118bb11bb124dc24dc8b11a8f7b20a131"
        },
        {
          "signature": "c5c0e561f878cf4ab12a0f45db4943fac22316997e79bc225772750dfdf9982132deea52336c8575e9ff39772e75092c8fd64b5518d4f6cc722afa18843b930b",
          "voterVK": "FVT9+EybM5YFJKmYZoUa0hb6mZEZXmtChhxwpTS/Gn8QeIaDzpFBq889IXTCt5Idew+JT+9f+tiXAluAlCNtng==",
          "proposalId": "530633b209d8c29c65306e5c6b615f0118bb11bb124dc24dc8b11a8f7b20a131"
        },
        {
          "signature": "efe8a9b511c33829ec356932515c9dadafd9fd918ba72d448c58c4f78960fe41978a3876a0d7057f5e93c2339c6689eb5d77a2842c363d358ae7af0f72a51305",
          "voterVK": "HV5+gKgoYbGoAB34NMkIDvKBrulsRyfCZmS0Fizrl7bUnEfMs9/EyvLcgDaqLdfNLNfFzBSgAR1QWNWADQ0Gig==",
          "proposalId": "530633b209d8c29c65306e5c6b615f0118bb11bb124dc24dc8b11a8f7b20a131"
        },
        {
          "signature": "2fc599536b8b9b2c8f2cd64e39587c29439a6baf16b05f577b6fc03741662793e6b97e67ebdda77e6aeefa944fa2dd073bc755dde4ee0b97c37e6d0d19fb3201",
          "voterVK": "9nyylndB1x0liSdnz0LhpoufvOpEuZ1jJLcrDZYspFk/mj10EkWZUCi/wfiST0hbeunKp2lhx+HbGpNuMlEvEA==",
          "proposalId": "530633b209d8c29c65306e5c6b615f0118bb11bb124dc24dc8b11a8f7b20a131"
        }
      ],
      "proposal": {
        "signature": "d4b2cba5657e385aace878bf0678e02f3953cf545665fc58f1e3a1612a86aab6a19af3de67ef12ecdb94b56baee7fdc1fa1bbbc49cd6a00da42ba2723bbe900e",
        "body": {
          "protocolVersion": {
            "minor": 1,
            "major": 0,
            "patch": 0
          },
          "parametersUpdate": {
            "txFeePolicy": null,
            "maxTxSize": null,
            "scriptVersion": null,
            "updateProposalTheshold": null,
            "maxBlockSize": null,
            "maxHeaderSize": null,
            "mpcThreshold": null,
            "slotDuration": null,
            "maxProposalSize": null,
            "softforkRule": null,
            "heavyDlgThreshold": null,
            "updateVoteThreshold": null,
            "unlockStakeEpoch": null,
            "updateProposalTTL": null
          },
          "metadata": {
            "win64": "2212502699039a859989f1ba1b1d758b21651690c49dff048394936e207d5d62",
            "macos64": "21398c0a46654d56e78e6cbf84ab4f8c4599ce0eb11c7193ae2a2a71ac3399a8"
          },
          "softwareVersion": {
            "appName": "csl-daedalus",
            "number": 10
          }
        },
        "issuer": "LxnxuG5pTlojgG7TCjSsrrQTEWIpd5Lpakd/Jlmh3J1N5vIPAlLn5W8WqnSTj94W+OhjYzD3QN5vWJqDwqA8Sw=="
      }
    },
    "txPayload": [],
    "dlgPayload": []
  },
  "header": {
    "signature": {
      "signature": "c5a9c90b65f2f7a11ae93c8a784554422734432b49e7185933eee450eba09739d16c9f29478f0d30cf20988926d366c5e21f3d09b3f757fc20b8d00fa2cb3b0b",
      "dlgCertificate": {
        "signature": "cb4e69ff882552865e93312fd00970bc7ab36296de0c50f50e4a7e0ce4b6cb0eb9a55794e20b670426f35cfbcd5f3368f9de5c403677fafe408e6a92eb5f4a00",
        "delegateVK": "9nyylndB1x0liSdnz0LhpoufvOpEuZ1jJLcrDZYspFk/mj10EkWZUCi/wfiST0hbeunKp2lhx+HbGpNuMlEvEA==",
        "issuerVK": "b0BEmpRyqoitufH0FAJIiTqpaDQYgpJ0uMuiPO3reI1AiAZD5fkvu8o9vWl924+VVwe7UEIRbOGCHk/o6Vel3g==",
        "epoch": 0
      }
    },
    "proof": {
      "utxo": {
        "root": "0e5751c026e543b2e8ab2eb06099daa1d1e5df47778f7787faab45cdf12fe3a8",
        "witnessesHash": "afc0da64183bf2664f3d4eec7238d524ba607faeeab24fc100eb861dba69971b",
        "number": 0
      },
      "delegation": "afc0da64183bf2664f3d4eec7238d524ba607faeeab24fc100eb861dba69971b",
      "update": "bf400c13101fd7c6ea8f017a445fb82f312942eefdee0e88e54c226642ef5147"
    },
    "protocolVersion": {
      "minor": 1,
      "major": 0,
      "patch": 0
    },
    "genesisKey": "b0BEmpRyqoitufH0FAJIiTqpaDQYgpJ0uMuiPO3reI1AiAZD5fkvu8o9vWl924+VVwe7UEIRbOGCHk/o6Vel3g==",
    "protocolMagicId": 633343913,
    "slot": 969188,
    "softwareVersion": {
      "appName": "cardano-sl",
      "number": 0
    },
    "prevHash": "130696646fa875d1fd46a438c7e34c089b88fe75ce83dd6d8b4cf8814892977e",
    "blockHeight": 966804
  }
}

[dcoutts]

Fixed by IntersectMBO/cardano-ledger#768