[Staking] Patch delegations total mismatch

[4meta5]

What does it do?

MOON-1517

• unit tests that fail before changes and pass after
• migration to fix incorrect state (by resumming all the delegations for all candidates)
• unit test for migration
• add migration to migrations config

Summary of Bug

The totalTopDelegations value was not updated after delegator_bond_more which means that total_counted = collator_bond + totalTopDelegations was not correctly updated and total_counted is used to determine which candidates are chosen in the active set.

Consequences

This bug does not enable abuse AFAICT. It does mean some data is incorrect, but the runtime migration fixes it.

It will understate the total_counted for collators that have delegations which were increased via delegator_bond_more. Moreover, it will overestimate each delegation share when paying out rewards leading to increased inflation rewards. Likewise, the patch needs to be applied ASAP to minimize the impact on reward payouts.

[girazoki]

Same comment here as before, what happens if its bottom?

[4meta5]

if its just bottom then total_changed == false so it would never enter this if branch

[girazoki]

Unrelated, but can you comment the xcm migrations that have been applied in 1200 too? I think this PR might be a good opportunity to do so

[girazoki]

@4meta5 did you check with try-runtime?

[girazoki]

I recall @notlesh said try-runtime was broken

[4meta5]

It compiles with the try-runtime flag, but I'm not sure how to use it to test because the state needs to be corrupted before the migration is called. It was more direct do this in the unit test context because I could set the state directly to corrupt it like what the bug caused.

[librelois]

It compiles with the try-runtime flag, but I'm not sure how to use it to test because the state needs to be corrupted before the migration is called. It was more direct do this in the unit test context because I could set the state directly to corrupt it like what the bug caused.

The purpose of try-runtime is to be able to download a storage of a live chain for testing on real data.

To test the migration you can use the following command:

./target/release/moonbeam try-runtime on-runtime-upgrade live -u <rpc_endpoint> -a <block_hash> -p ParachainStaking,Migrations

Thus, try-runtime will download only the storage entries for the listed pallets instances.

[girazoki]

I think for what @librelois wants to do you need to have a node synched (right @notlesh ?). If so, what you can is get the state of the corrupted storage in one of the chains for an account (with the RPC getStorage, by inserting the key of the corrupted storage).

Then you can launch locally a network, and use sudo.setStorage, same key as before, and you will have your corrupted storage there, and you can pass your migration to your local network

[librelois]

I think for what @librelois wants to do you need to have a node synched. If so, what you can is get the state of the corrupted storage in one of the chains for an account (with the RPC getStorage, by inserting the key of the corrupted storage).

Then you can launch locally a network, and use sudo.setStorage, same key as before, and you will have your corrupted storage there, and you can pass your migration to your local network

No, with try-runtime you don't need any node synced

[4meta5]

I ran it on moonbase and try-runtime executed without error, logs look like it worked and did fix a few candidate total_counted

Command and output

[crystalin]

@4meta5 can you get the PoV size also with try-runtime ?

[4meta5]

@crystalin for moonbase, it is included in the bottom of the output https://gist.github.com/4meta5/7ef1357d2ceca43476566284ea362856

2022-02-18 10:08:56 proof size: 1.73 MB (1772.384765625 KB) (1814922 bytes)    
2022-02-18 10:08:56 compact proof size: 1.72 MB (1756.1943359375 KB) (1798343 bytes)    
2022-02-18 10:08:56 zstd-compressed compact proof 1.45 MB (1482.875 KB) (1518464 bytes)    
2022-02-18 10:08:56 TryRuntime_on_runtime_upgrade executed without errors. Consumed weight = 186050000000, total weight = 500000000000 (0.3721)

for moonriver/moonbeam, try-runtime does not yet work (MOON-1200 is the ticket to update it)