Error on replay with wrong app hash calculation (block #914381)

[cyborgshead]

Current Behavior

Cyberd validator node stack on block 914381 with error on replay and wrong app hash

Expected Behavior

Continuing of sync and get online validation

Reproduction

Ubuntu 18.04.01 LTS/NVIDIA-SMI 415.27/Driver Version: 415.27/CUDA Version: 10.0
i7-7700K @ 4.20GHz/ 32GB RAM / MSI GTX 1070 8GB / 512 SSD

Log

panic: Panicked questionably: Failed to process committed block (914381:9F818FD2149C6CC6E52AF7A4932734D09732BDDEAB629D699B8888BDC0CD1C4E): Wrong Block.Header.AppHash.  Expected 64D2D5DF2A679423AACB803DFAF79D8BA3BB224919BD5FC034A491C9075E8283, got 292590BC3F2700FBF3C47178DD37638118A92E720AE03C5FA0D546D1C7B29388

goroutine 529363 [running]:
github.com/tendermint/tendermint/libs/common.PanicQ(0xe2bc20, 0xc00d5b8520)
	/root/go/pkg/mod/github.com/tendermint/tendermint@v0.27.4/libs/common/errors.go:245 +0xb8
github.com/tendermint/tendermint/blockchain.(*BlockchainReactor).poolRoutine(0xc0000f6680)
	/root/go/pkg/mod/github.com/tendermint/tendermint@v0.27.4/blockchain/reactor.go:341 +0x1033
created by github.com/tendermint/tendermint/blockchain.(*BlockchainReactor).OnStart
	/root/go/pkg/mod/github.com/tendermint/tendermint@v0.27.4/blockchain/reactor.go:118 +0x89
ERROR: Error during handshake: Error on replay: Wrong Block.Header.AppHash.  Expected 64D2D5DF2A679423AACB803DFAF79D8BA3BB224919BD5FC034A491C9075E8283, got 292590BC3F2700FBF3C47178DD37638118A92E720AE03C5FA0D546D1C7B29388
ERROR: Error during handshake: Error on replay: Wrong Block.Header.AppHash.  Expected 64D2D5DF2A679423AACB803DFAF79D8BA3BB224919BD5FC034A491C9075E8283, got 292590BC3F2700FBF3C47178DD37638118A92E720AE03C5FA0D546D1C7B29388
ERROR: Error during handshake: Error on replay: Wrong Block.Header.AppHash.  Expected 64D2D5DF2A679423AACB803DFAF79D8BA3BB224919BD5FC034A491C9075E8283, got 292590BC3F2700FBF3C47178DD37638118A92E720AE03C5FA0D5

Additional Information

[hleb-albau]

@litvintech yeah, reproduced. Currently, we don't know exactly bug, need more investigation.
But, already knowing that this happens on soft fork block. Seems, some hidden bug on index loading after reload.

[hleb-albau]

@litvintech @xhipster @arturalbov Bad news, there is a bug, that causes all full nodes joining network right now to panic at 914381. Bug related, to our wrong assumption of stake update in cosmos-sdk. We miss user stake update for jailing validators and calculate the wrong rank using wrong user stake. There is no easy way to eliminate it for current testnet. So, I suggest focussing on fixing the bug and moving to #205.
There is no easy way to track bounded stake in cosmos right now, so maybe for 0.1.1 we should change the rank calculation to honor only NotBounded user stake, and investigate ways to archive bounded +unboded stake for next testnets.

[cyborgshead]

On Mars I have this

I[28016-01-28|10:23:21.552] Starting ABCI with Tendermint                module=main 


I[28016-01-28|10:23:21.584] Loading mem state                            module=main 


I[28016-01-28|10:23:41.161] App loaded                                   module=main time=19.576980759s


I[28016-01-28|10:23:41.323] Rank calculated                              module=main time=154.813899ms links=38 cids=45


panic: version 914360 was already saved to different hash 48C6928913C3A60C0DDC36B803E622128ABBD807DF8DFACFB80B3BD742A4E002 (existing hash D7AE39ADE93FE78AC5BDE6E6956C5C026D8B79532A79F337B398419B5C8CEAAC)




goroutine 1 [running]:


github.com/cosmos/cosmos-sdk/store.(*iavlStore).Commit(0xc00018cc40, 0xfb2b5f, 0x5, 0x20, 0x20)


	/root/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.29.0/store/iavlstore.go:75 +0x1dd


github.com/cosmos/cosmos-sdk/store.commitStores(0xdf3b8, 0xc00020d620, 0x7020103, 0x0, 0xffffffffffffffff, 0xc007a5d928)


	/root/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.29.0/store/rootmultistore.go:488 +0x177


github.com/cosmos/cosmos-sdk/store.(*rootMultiStore).Commit(0xc0004bd490, 0xc0085acdb0, 0xc0040ba6c0, 0xd, 0x8)


	/root/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.29.0/store/rootmultistore.go:197 +0x47


github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).Commit(0xc000b46800, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)


	/root/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.29.0/baseapp/baseapp.go:814 +0xea


github.com/cybercongress/cyberd/app.(*CyberdApp).Commit(0xc000202500, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)


	/sources/app/app.go:544 +0x50


github.com/tendermint/tendermint/abci/client.(*localClient).CommitSync(0xc0082d6180, 0x0, 0x0, 0x0)


	/root/go/pkg/mod/github.com/tendermint/tendermint@v0.27.4/abci/client/local_client.go:215 +0xc0


github.com/tendermint/tendermint/proxy.(*appConnConsensus).CommitSync(0xc00093e8c0, 0xc003428fa0, 0x121c340, 0xc00093e8c0)


	/root/go/pkg/mod/github.com/tendermint/tendermint@v0.27.4/proxy/app_conn.go:81 +0x33


github.com/tendermint/tendermint/state.ExecCommitBlock(0x121c340, 0xc00093e8c0, 0xc00082a3c0, 0x12176c0, 0xc003428fa0, 0xc0085f03f0, 0x1220b80, 0xc002f00fa0, 0x14, 0x400000, ...)


	/root/go/pkg/mod/github.com/tendermint/tendermint@v0.27.4/state/execution.go:522 +0x27e


github.com/tendermint/tendermint/consensus.(*Handshaker).replayBlocks(0xc003374840, 0x8, 0x0, 0xc007b644a0, 0x6, 0xc007b644ab, 0x5, 0xdf3ca, 0x45, 0xc0083f2600, ...)


	/root/go/pkg/mod/github.com/tendermint/tendermint@v0.27.4/consensus/replay.go:409 +0x244


github.com/tendermint/tendermint/consensus.(*Handshaker).ReplayBlocks(0xc003374840, 0x8, 0x0, 0xc007b644a0, 0x6, 0xc007b644ab, 0x5, 0xdf3ca, 0x45, 0xc0083f2600, ...)


	/root/go/pkg/mod/github.com/tendermint/tendermint@v0.27.4/consensus/replay.go:347 +0x880


github.com/tendermint/tendermint/consensus.(*Handshaker).Handshake(0xc003374840, 0x1221a00, 0xc008398230, 0x2, 0x12176c0)


	/root/go/pkg/mod/github.com/tendermint/tendermint@v0.27.4/consensus/replay.go:253 +0x516


github.com/tendermint/tendermint/node.NewNode(0xc0001599e0, 0x1217880, 0xc002fb14a0, 0xc0026bdac0, 0x120bbc0, 0xc004b1ee40, 0xc0026bdce0, 0x11262d0, 0xc0026bdcf0, 0x12176c0, ...)


	/root/go/pkg/mod/github.com/tendermint/tendermint@v0.27.4/node/node.go:204 +0x6cf


github.com/cosmos/cosmos-sdk/server.startInProcess(0xc000199ac0, 0x1126ad0, 0x1d, 0x0, 0x0)


	/root/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.29.0/server/start.go:118 +0x3f1


github.com/cosmos/cosmos-sdk/server.StartCmd.func1(0xc000b44d80, 0x1bc8300, 0x0, 0x0, 0x0, 0x0)


	/root/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.29.0/server/start.go:40 +0xab


github.com/spf13/cobra.(*Command).execute(0xc000b44d80, 0x1bc8300, 0x0, 0x0, 0xc000b44d80, 0x1bc8300)


	/root/go/pkg/mod/github.com/spf13/cobra@v0.0.1/command.go:698 +0x47a


github.com/spf13/cobra.(*Command).ExecuteC(0xc0000d2d80, 0xc000199d80, 0xc000495e40, 0xc2773d)


	/root/go/pkg/mod/github.com/spf13/cobra@v0.0.1/command.go:783 +0x2dc


github.com/spf13/cobra.(*Command).Execute(0xc0000d2d80, 0xc26ff2, 0xc000187610)


	/root/go/pkg/mod/github.com/spf13/cobra@v0.0.1/command.go:736 +0x2b


github.com/tendermint/tendermint/libs/cli.Executor.Execute(0xc0000d2d80, 0x1126ef8, 0x3, 0xc000b26e50)


	/root/go/pkg/mod/github.com/tendermint/tendermint@v0.27.4/libs/cli/setup.go:89 +0x4e


main.main()


	/sources/daemon/main.go:63 +0x53d

[hleb-albau]

Fixed in 0.1.1 network.