Problem: non-determinism in reward distribution

[tomtau]

(being investigated on 0.5)
some nodes randomly after reward distribution ended up with a different app hash

[yihuang]

current observations from logs, where issue happens:

• minted amount and total staking is good.
• but the staking diffs returned by the begin_block is inconsistent.

Two possible reasons for that:

1. period_bonus is already inconsistent.
2. period_bonus is good, but the reward distribution code is non-determinism.

By reading the code, I think the second case is quite unlikely.
Currently we added more logs to try to rule out the first case.
If it's the first case, then problem space further split into two directions:

1. fee accumulation non-determinism
2. slashing accumulation non-determinism

[tomtau]

for fee accumulation, it's only here:
https://github.com/crypto-com/chain/blob/release/v0.5/chain-abci/src/app/mod.rs#L239
and the TX order should be the same / deterministic (unless something strange happens with tokio async reading/writing on the socket: https://github.com/tendermint/rust-abci/blob/develop/src/server.rs#L38 )