Parachain ValidationFunctionApplied different relay block than Relay

[crystalin]

Moonriver (para id 2023) did a runtime upgrade, Friday around 2:45am UTC
The runtime was planned to be applied 1 hour later.

When the time came, the runtime upgrade was applied on the parachain.
The collators started to produce blocks with the new runtime successfully (no error in the logs neither), but the relay validator refused them (Digest item must match that calculated.).
The parachain became stalled at that point with no possibility of recovery.

After looking at the paras state, we observed that the relay paras.futureCodeUpgrades was planned for block 8,263,579 (which was already past when looking at it)

but the parachain parachainSystem.ValidationFunctionStored reported the execution would happen on relay block 8,263,577
https://polkadot.js.org/apps/?rpc=wss%3A%2F%2Fwss.moonriver.moonbeam.network#/explorer/query/0x62e6661e7adc691420fdc125be711be59b91aaa7fc4e6976e3cd3e408464df84

And the parachainSystem.ValidationFunctionApplied reported the execution happened on block 8,263,578
(https://polkadot.js.org/apps/?rpc=wss%3A%2F%2Fwss.moonriver.moonbeam.network#/explorer/query/0xef9af479d1bf580fa9289d7bdd331f7d11056e398448c2e888a9eb69224d5c27)

According to @bkchr, this is really really bad
:(

(edit)
The code hashes:

paras.futureCodeHash: 0x0180ee63bf0c715462bfcdd8de7072ff462af9ce99226a06b79bb1a581e71449 (spec_version 155)
paras.currentCodeHash: 0x08f6d9aa9cc1f326cf38e3385fe56de65700bc42a3d84350a2224f0d798e18d4 (spec_version 53)

runtime-53 (moonriver) can be found here: https://github.com/PureStake/moonbeam/releases/tag/v0.8.4
runtime-155 (moonriver) can be found here: https://github.com/PureStake/moonbeam/releases/tag/v0.9.2

[xlc]

https://moonriver.subscan.io/block/67640

The parachainSystem.setValidationData sets the relaychain block number to 8262979.

This runtime upgrade is performed using system.setCode via democracy. This means it is called in on_initialize hook, that is executed before inherents. So it is reading the old relaychain block number instead of the new one.

We really need #312 to address this.

Meanwhile, parachain team should be able to use parachainSystem.authorizeUpgrade for democracy and use parachainSystem.enactAuthorizedUpgrade to actually perform the upgrade.