Node failed with db corruption error

[mrlp4]

daemon enabled as systemd service, through cosmosd

Log

дек 23 11:18:16 immortal cosmovisor[213782]: I[2020-12-23|11:18:16.752] Executed block                               module=state height=3662964 validTxs=0 invalidTxs=0
дек 23 11:18:16 immortal cosmovisor[213782]: I[2020-12-23|11:18:16.770] Committed state                              module=state height=3662964 txs=0 appHash=F4744B14C7DDF5F86ED1DE139C843AB91384848370B5616E152E5B5E09AF25CB
дек 23 11:18:22 immortal cosmovisor[213782]: I[2020-12-23|11:18:22.345] Executed block                               module=state height=3662965 validTxs=0 invalidTxs=0
дек 23 11:18:22 immortal cosmovisor[213782]: E[2020-12-23|11:18:22.356] CONSENSUS FAILURE!!!                         module=consensus err="error in snapShotBatch write: leveldb/table: corruption on data-block (pos=322643): checksum mismatch, want=0x322c5338 got=0x141540ef [file=95565363.ldb]" stack="goroutine 4573163 [running]:\nruntime/debug.Stack(0xc000dddba8, 0x12f51a0, 0xc252046680)\n\t/usr/local/go/src/runtime/debug/stack.go:24 +0x9d\ngithub.com/tendermint/tendermint/consensus.(*State).receiveRoutine.func2(0xc0004dce00, 0x15afe30)\n\t/root/go/pkg/mod/github.com/tendermint/tendermint@v0.33.6/consensus/state.go:617 +0x57\npanic(0x12f51a0, 0xc252046680)\n\t/usr/local/go/src/runtime/panic.go:969 +0x166\ngithub.com/cosmos/cosmos-sdk/store/iavl.(*Store).Commit(0xc00d5402a0, 0xc000ddddf0, 0x415e34, 0xc000dddee8, 0xc020bad400)\n\t/root/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.38.5/store/iavl/store.go:127 +0x1ec\ngithub.com/cosmos/cosmos-sdk/store/rootmulti.commitStores(0x37e475, 0xc001a77020, 0xc2613860f0, 0xc261386120, 0xc001a76150, 0x0)\n\t/root/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.38.5/store/rootmulti/store.go:634 +0x165\ngithub.com/cosmos/cosmos-sdk/store/rootmulti.(*Store).Commit(0xc0001d61b0, 0xc00c50ade0, 0x17f1aa0, 0xc015855a40, 0xc000dde010)\n\t/root/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.38.5/store/rootmulti/store.go:292 +0x44\ngithub.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).Commit(0xc000ca6280, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)\n\t/root/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.38.5/baseapp/abci.go:235 +0x17e\ngithub.com/cybercongress/go-cyber/app.(*CyberdApp).Commit(0xc000cb0e00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)\n\t/root/go-cyber/app/app.go:562 +0x4f\ngithub.com/tendermint/tendermint/abci/client.(*localClient).CommitSync(0xc2245fe120, 0x0, 0x0, 0x0)\n\t/root/go/pkg/mod/github.com/tendermint/tendermint@v0.33.6/abci/client/local_client.go:215 +0xcc\ngithub.com/tendermint/tendermint/proxy.(*appConnConsensus).CommitSync(0xc0e5cd8940, 0x0, 0x0, 0xa)\n\t/root/go/pkg/mod/github.com/tendermint/tendermint@v0.33.6/proxy/app_conn.go:81 +0x33\ngithub.com/tendermint/tendermint/state.(*BlockExecutor).Commit(0xc1655942a0, 0xa, 0x0, 0xc010dffd30, 0x6, 0xc010dffd40, 0x7, 0x37e475, 0xc14895a380, 0x20, ...)\n\t/root/go/pkg/mod/github.com/tendermint/tendermint@v0.33.6/state/execution.go:216 +0x246\ngithub.com/tendermint/tendermint/state.(*BlockExecutor).ApplyBlock(0xc1655942a0, 0xa, 0x0, 0xc010dffd30, 0x6, 0xc010dffd40, 0x7, 0x37e475, 0xc14895a380, 0x20, ...)\n\t/root/go/pkg/mod/github.com/tendermint/tendermint@v0.33.6/state/execution.go:170 +0x63d\ngithub.com/tendermint/tendermint/consensus.(*State).finalizeCommit(0xc0004dce00, 0x37e475)\n\t/root/go/pkg/mod/github.com/tendermint/tendermint@v0.33.6/consensus/state.go:1459 +0x8f5\ngithub.com/tendermint/tendermint/consensus.(*State).tryFinalizeCommit(0xc0004dce00, 0x37e475)\n\t/root/go/pkg/mod/github.com/tendermint/tendermint@v0.33.6/consensus/state.go:1377 +0x383\ngithub.com/tendermint/tendermint/consensus.(*State).enterCommit.func1(0xc0004dce00, 0x0, 0x37e475)\n\t/root/go/pkg/mod/github.com/tendermint/tendermint@v0.33.6/consensus/state.go:1312 +0x90\ngithub.com/tendermint/tendermint/consensus.(*State).enterCommit(0xc0004dce00, 0x37e475, 0x0)\n\t/root/go/pkg/mod/github.com/tendermint/tendermint@v0.33.6/consensus/state.go:1349 +0x61e\ngithub.com/tendermint/tendermint/consensus.(*State).addVote(0xc0004dce00, 0xc0e8579cc0, 0xc16945af00, 0x28, 0x0, 0x120, 0x120a240)\n\t/root/go/pkg/mod/github.com/tendermint/tendermint@v0.33.6/consensus/state.go:1891 +0xa39\ngithub.com/tendermint/tendermint/consensus.(*State).tryAddVote(0xc0004dce00, 0xc0e8579cc0, 0xc16945af00, 0x28, 0xc00d8c6b40, 0xc00d8c6c60, 0x30)\n\t/root/go/pkg/mod/github.com/tendermint/tendermint@v0.33.6/consensus/state.go:1707 +0x67\ngithub.com/tendermint/tendermint/consensus.(*State).handleMsg(0xc0004dce00, 0x17d5d80, 0xc11fd31940, 0xc16945af00, 0x28)\n\t/root/go/pkg/mod/github.com/tendermint/tendermint@v0.33.6/consensus/state.go:713 +0x525\ngithub.com/tendermint/tendermint/consensus.(*State).receiveRoutine(0xc0004dce00, 0x0)\n\t/root/go/pkg/mod/github.com/tendermint/tendermint@v0.33.6/consensus/state.go:648 +0x709\ncreated by github.com/tendermint/tendermint/consensus.(*State).OnStart\n\t/root/go/pkg/mod/github.com/tendermint/tendermint@v0.33.6/consensus/state.go:335 +0x13a\n"

Node Immortal

[mrlp4]

Another node catch the same issue:

Dec 23 17:00:06 callisto cosmosd[8060]: I[2020-12-23|17:00:06.538] Executed block                               module=state height=3666525 validTxs=0 invalidTxs=0
Dec 23 17:00:06 callisto cosmosd[8060]: I[2020-12-23|17:00:06.550] Committed state                              module=state height=3666525 txs=0 appHash=04FB9BC7287961795065C83E570071FC18C5DFCC023B2949C$
Dec 23 17:00:12 callisto cosmosd[8060]: I[2020-12-23|17:00:12.055] Executed block                               module=state height=3666526 validTxs=0 invalidTxs=0
Dec 23 17:00:12 callisto cosmosd[8060]: I[2020-12-23|17:00:12.066] Committed state                              module=state height=3666526 txs=0 appHash=04FB9BC7287961795065C83E570071FC18C5DFCC023B2949C$
Dec 23 17:00:17 callisto cosmosd[8060]: I[2020-12-23|17:00:17.871] Executed block                               module=state height=3666527 validTxs=0 invalidTxs=0
Dec 23 17:00:17 callisto cosmosd[8060]: E[2020-12-23|17:00:17.879] CONSENSUS FAILURE!!!                         module=consensus err="error in snapShotBatch write: leveldb/table: corruption on data-block (pos=992712): checksum mismatch, want=0x9e76a845 got=0x13b9fd28 [file=74099550.ldb]" stack="goroutine 4452692 [running]:\nruntime/debug.Stack(0xc001375bb8, 0x12f1ea0, 0xc262cd5520)\n\t/root/.gvm/gos/go1.14/src/runtime/debug/stack.go:24 +0x9d\ngithub.com/tendermint/tendermint/consensus.(*State).receiveRoutine.func2(0xc000500a80, 0x15ac910)\n\t/root/.gvm/pkgsets/go1.14/global/pkg/mod/github.com/tendermint/tendermint@v0.33.6/consensus/state.go:617 +0x57\npanic(0x12f1ea0, 0xc262cd5520)\n\t/root/.gvm/gos/go1.14/src/runtime/panic.go:967 +0x15d\ngithub.com/cosmos/cosmos-sdk/store/iavl.(*Store).Commit(0xc0447276e0, 0xc001375df0, 0x4151b4, 0xc001375ee8, 0xc01c948400)\n\t/root/.gvm/pkgsets/go1.14/global/pkg/mod/github.com/cosmos/cosmos-sdk@v0.38.5/store/iavl/store.go:127 +0x1ec\ngithub.com/cosmos/cosmos-sdk/store/rootmulti.commitStores(0x37f25f, 0xc005500480, 0xc1cdab2600, 0xc1cdab2630, 0xc0054688d0, 0x0)\n\t/root/.gvm/pkgsets/go1.14/global/pkg/mod/github.com/cosmos/cosmos-sdk@v0.38.5/store/rootmulti/store.go:634 +0x165\ngithub.com/cosmos/cosmos-sdk/store/rootmulti.(*Store).Commit(0xc000520510, 0xc00e781ba0, 0x17ee540, 0xc00b058b40, 0x7ff4fb202e00)\n\t/root/.gvm/pkgsets/go1.14/global/pkg/mod/github.com/cosmos/cosmos-sdk@v0.38.5/store/rootmulti/store.go:292 +0x44\ngithub.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).Commit(0xc001165a40, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)\n\t/root/.gvm/pkgsets/go1.14/global/pkg/mod/github.com/cosmos/cosmos-sdk@v0.38.5/baseapp/abci.go:235 +0x17e\ngithub.com/cybercongress/go-cyber/app.(*CyberdApp).Commit(0xc000e16e00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)\n\t/root/go-cyber/app/app.go:562 +0x4f\ngithub.com/tendermint/tendermint/abci/client.(*localClient).CommitSync(0xc2bd3dc120, 0x0, 0x0, 0x0)\n\t/root/.gvm/pkgsets/go1.14/global/pkg/mod/github.com/tendermint/tendermint@v0.33.6/abci/client/local_client.go:215 +0xcc\ngithub.com/tendermint/tendermint/proxy.(*appConnConsensus).CommitSync(0xc000017920, 0x0, 0x0, 0xa)\n\t/root/.gvm/pkgsets/go1.14/global/pkg/mod/github.com/tendermint/tendermint@v0.33.6/proxy/app_conn.go:81 +0x33\ngithub.com/tendermint/tendermint/state.(*BlockExecutor).Commit(0xc15fd3c000, 0xa, 0x0, 0xc1fade5830, 0x6, 0xc1fade5840, 0x7, 0x37f25f, 0xc282e3e300, 0x20, ...)\n\t/root/.gvm/pkgsets/go1.14/global/pkg/mod/github.com/tendermint/tendermint@v0.33.6/state/execution.go:216 +0x246\ngithub.com/tendermint/tendermint/state.(*BlockExecutor).ApplyBlock(0xc15fd3c000, 0xa, 0x0, 0xc1fade5830, 0x6, 0xc1fade5840, 0x7, 0x37f25f, 0xc282e3e300, 0x20, ...)\n\t/root/.gvm/pkgsets/go1.14/global/pkg/mod/github.com/tendermint/tendermint@v0.33.6/state/execution.go:170 +0x63d\ngithub.com/tendermint/tendermint/consensus.(*State).finalizeCommit(0xc000500a80, 0x37f25f)\n\t/root/.gvm/pkgsets/go1.14/global/pkg/mod/github.com/tendermint/tendermint@v0.33.6/consensus/state.go:1459 +0x8f5\ngithub.com/tendermint/tendermint/consensus.(*State).tryFinalizeCommit(0xc000500a80, 0x37f25f)\n\t/root/.gvm/pkgsets/go1.14/global/pkg/mod/github.com/tendermint/tendermint@v0.33.6/consensus/state.go:1377 +0x383\ngithub.com/tendermint/tendermint/consensus.(*State).enterCommit.func1(0xc000500a80, 0x0, 0x37f25f)\n\t/root/.gvm/pkgsets/go1.14/global/pkg/mod/github.com/tendermint/tendermint@v0.33.6/consensus/state.go:1312 +0x90\ngithub.com/tendermint/tendermint/consensus.(*State).enterCommit(0xc000500a80, 0x37f25f, 0x0)\n\t/root/.gvm/pkgsets/go1.14/global/pkg/mod/github.com/tendermint/tendermint@v0.33.6/consensus/state.go:1349 +0x61e\ngithub.com/tendermint/tendermint/consensus.(*State).addVote(0xc000500a80, 0xc00e192fa0, 0xc17166c270, 0x28, 0x0, 0x120, 0x1206fa0)\n\t/root/.gvm/pkgsets/go1.14/global/pkg/mod/github.com/tendermint/tendermint@v0.33.6/consensus/state.go:1891 +0xa39\ngithub.com/tendermint/tendermint/consensus.(*State).tryAddVote(0xc000500a80, 0xc00e192fa0, 0xc17166c270, 0x28, 0xc00e3a7d40, 0xc00e3a7e60, 0x30)\n\t/root/.gvm/pkgsets/go1.14/global/pkg/mod/github.com/tendermint/tendermint@v0.33.6/consensus/state.go:1707 +0x67\ngithub.com/tendermint/tendermint/consensus.(*State).handleMsg(0xc000500a80, 0x17d2840, 0xc13214f558, 0xc17166c270, 0x28)\n\t/root/.gvm/pkgsets/go1.14/global/pkg/mod/github.com/tendermint/tendermint@v0.33.6/consensus/state.go:713 +0x525\ngithub.com/tendermint/tendermint/consensus.(*State).receiveRoutine(0xc000500a80, 0x0)\n\t/root/.gvm/pkgsets/go1.14/global/pkg/mod/github.com/tendermint/tendermint@v0.33.6/consensus/state.go:648 +0x709\ncreated by github.com/tendermint/tendermint/consensus.(*State).OnStart\n\t/root/.gvm/pkgsets/go1.14/global/pkg/mod/github.com/tendermint/tendermint@v0.33.6/consensus/state.go:335 +0x13a\n"
Dec 23 17:00:24 callisto cosmosd[8060]: E[2020-12-23|17:00:24.153] MConnection flush failed                     module=p2p peer=83f1badf90fb5fdbc04cef4d7531bf97b705dd3c@176.37.214.146:20663 err="write tc$

Node litvintech