Kusama v0.9.12 runtime upgrade at 9866422 broke mining

[h4x3rotab]

Symptom

1. Miners and Gatekeepers relying on pherry are stuck at Kusama block 9866422 with error:

[2021-10-29T16:04:11Z INFO  pherry] bridge() exited with error: Rpc error: Parse error: Decode error: Could not decode DigestItemType, variant doesn't exist

    Caused by:
        0: Parse error: Decode error: Could not decode DigestItemType, variant doesn't exist
        1: Decode error: Could not decode DigestItemType, variant doesn't exist

2. Even on PRB where the block can be processed, pRuntime will also fail to sync block with error:

[2021-10-29T16:00:14Z INFO  rocket::rocket] Matched: POST /bin_api/sync_combined_headers (sync_combined_headers)
[INFO  phactory::bin_api_service] {"payload":"{\"message\":\"Decode input parameter 

Affected

• Miners cannot mine (pherry or prb)
• Gatekeepers are stuck

Root cause

The new Kusama runtime introduces a new Digest Item, which is not decodable by the old block header definition (in both pruntime and subxt). pherry depends on subxt to decode the relay chain block header. pruntime depends on the raw block header definition to decode the inputs. Therefore when they panicked when 9866422 is reached.

Solution

The team is working on a mitigation and a long term solution. The ETA of the mitigation is Oct 30 16:00 (UTC).

The Phala team will release a new quick fix that carry with the new Digest Item.

[h4x3rotab]

Short term mitigation

Fortunately the substrate dependency is introduced by a git submodule. So we can cherry-pick the DigestItem commit and update it locally. Then we are going to release a new pherry (based on the master branch), and a new pruntime (based on the last pruntime release, because there are too many changes not well-tested on the master branch), with the digest item commit.

Branches:

• kusama-9122-quickfix-pherry
• kusama-9122-quickfix-pruntime

New pruntime hash:

• 0x199f62feaba1c4ca6b801a9ffaee687206dde93421151f6f02d903b0c88b221a00000000815f42f11cf64430c30bab7816ba596a1da0130c3b028b673133a66cf9a3e0e6
• Proposed as Motion ? (link?)

[h4x3rotab]

Already upgraded v0.9.12(13) and with Metadata V14.