New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Wrong app hash calculated after node restart. #453
Comments
Issue was submitted by 3 more times by independent people. Hash calculated incorrectly after container/node restart. |
New issue catch. Node - Mars. docker stop euler-5
cp -r /euler-5/cyberd/data/ /media/mars/raid1/data.backups.euler5/04_02_2020/
docker start euler-5 Logs: root@mars:~# docker logs euler-5 --follow --tail 20
I[2020-02-04|08:55:53.339] Executed block module=state height=707840 validTxs=0 invalidTxs=0
I[2020-02-04|08:55:53.366] Committed state module=state height=707840 txs=0 appHash=A9E64B435B8DDD16C3BA9E043313865FC1BE552E362DB34180FC53A5858BD9E7
I[2020-02-04|08:55:53.382] Rank: data for gpu preparation module=main time=42.301204ms
I[2020-02-04|08:55:53.398] Rank: gpu calculations module=main time=16.491453ms
I[2020-02-04|08:55:53.399] Rank constructing tree module=main time=685.021µs
I[2020-02-04|08:55:53.399] Rank calculated module=main time=59.684628ms links=614 cids=600 hash=61274586122914ba62f422c95e82fca865a0a0cd35d86f336d4b93a858faf20c
I[2020-02-04|08:55:59.232] Executed block module=state height=707841 validTxs=0 invalidTxs=0
I[2020-02-04|08:55:59.256] Committed state module=state height=707841 txs=0 appHash=A9E64B435B8DDD16C3BA9E043313865FC1BE552E362DB34180FC53A5858BD9E7
I[2020-02-04|08:56:04.652] Executed block module=state height=707842 validTxs=0 invalidTxs=0
I[2020-02-04|08:56:04.677] Committed state module=state height=707842 txs=0 appHash=A9E64B435B8DDD16C3BA9E043313865FC1BE552E362DB34180FC53A5858BD9E7
I[2020-02-04|08:56:10.325] Executed block module=state height=707843 validTxs=0 invalidTxs=0
I[2020-02-04|08:56:10.351] Committed state module=state height=707843 txs=0 appHash=A9E64B435B8DDD16C3BA9E043313865FC1BE552E362DB34180FC53A5858BD9E7
I[2020-02-04|08:56:15.829] Executed block module=state height=707844 validTxs=0 invalidTxs=0
I[2020-02-04|08:56:15.859] Committed state module=state height=707844 txs=0 appHash=A9E64B435B8DDD16C3BA9E043313865FC1BE552E362DB34180FC53A5858BD9E7
I[2020-02-04|08:56:21.554] Executed block module=state height=707845 validTxs=0 invalidTxs=0
I[2020-02-04|08:56:21.582] Committed state module=state height=707845 txs=0 appHash=A9E64B435B8DDD16C3BA9E043313865FC1BE552E362DB34180FC53A5858BD9E7
I[2020-02-04|08:56:21.591] Rank: data for gpu preparation module=main time=36.306368ms
I[2020-02-04|08:56:21.607] Rank: gpu calculations module=main time=15.984518ms
I[2020-02-04|08:56:21.608] Rank constructing tree module=main time=635.017µs
I[2020-02-04|08:56:21.608] Rank calculated module=main time=53.115569ms links=614 cids=600 hash=61274586122914ba62f422c95e82fca865a0a0cd35d86f336d4b93a858faf20c
I[2020-02-04|08:56:26.899] Executed block module=state height=707846 validTxs=0 invalidTxs=0
I[2020-02-04|08:56:26.923] Committed state module=state height=707846 txs=0 appHash=A9E64B435B8DDD16C3BA9E043313865FC1BE552E362DB34180FC53A5858BD9E7
I[2020-02-04|08:56:32.568] Executed block module=state height=707847 validTxs=0 invalidTxs=0
I[2020-02-04|08:56:32.593] Committed state module=state height=707847 txs=0 appHash=A9E64B435B8DDD16C3BA9E043313865FC1BE552E362DB34180FC53A5858BD9E7
I[2020-02-04|08:56:38.224] Executed block module=state height=707848 validTxs=0 invalidTxs=0
I[2020-02-04|08:56:38.254] Committed state module=state height=707848 txs=0 appHash=A9E64B435B8DDD16C3BA9E043313865FC1BE552E362DB34180FC53A5858BD9E7
I[2020-02-04|08:56:43.835] Executed block module=state height=707849 validTxs=0 invalidTxs=0
I[2020-02-04|08:56:43.860] Committed state module=state height=707849 txs=0 appHash=A9E64B435B8DDD16C3BA9E043313865FC1BE552E362DB34180FC53A5858BD9E7
root@mars:~# docker logs euler-5 --follow --tail 20
I[2020-02-04|08:56:15.829] Executed block module=state height=707844 validTxs=0 invalidTxs=0
I[2020-02-04|08:56:15.859] Committed state module=state height=707844 txs=0 appHash=A9E64B435B8DDD16C3BA9E043313865FC1BE552E362DB34180FC53A5858BD9E7
I[2020-02-04|08:56:21.554] Executed block module=state height=707845 validTxs=0 invalidTxs=0
I[2020-02-04|08:56:21.582] Committed state module=state height=707845 txs=0 appHash=A9E64B435B8DDD16C3BA9E043313865FC1BE552E362DB34180FC53A5858BD9E7
I[2020-02-04|08:56:21.591] Rank: data for gpu preparation module=main time=36.306368ms
I[2020-02-04|08:56:21.607] Rank: gpu calculations module=main time=15.984518ms
I[2020-02-04|08:56:21.608] Rank constructing tree module=main time=635.017µs
I[2020-02-04|08:56:21.608] Rank calculated module=main time=53.115569ms links=614 cids=600 hash=61274586122914ba62f422c95e82fca865a0a0cd35d86f336d4b93a858faf20c
I[2020-02-04|08:56:26.899] Executed block module=state height=707846 validTxs=0 invalidTxs=0
I[2020-02-04|08:56:26.923] Committed state module=state height=707846 txs=0 appHash=A9E64B435B8DDD16C3BA9E043313865FC1BE552E362DB34180FC53A5858BD9E7
I[2020-02-04|08:56:32.568] Executed block module=state height=707847 validTxs=0 invalidTxs=0
I[2020-02-04|08:56:32.593] Committed state module=state height=707847 txs=0 appHash=A9E64B435B8DDD16C3BA9E043313865FC1BE552E362DB34180FC53A5858BD9E7
I[2020-02-04|08:56:38.224] Executed block module=state height=707848 validTxs=0 invalidTxs=0
I[2020-02-04|08:56:38.254] Committed state module=state height=707848 txs=0 appHash=A9E64B435B8DDD16C3BA9E043313865FC1BE552E362DB34180FC53A5858BD9E7
I[2020-02-04|08:56:43.835] Executed block module=state height=707849 validTxs=0 invalidTxs=0
I[2020-02-04|08:56:43.860] Committed state module=state height=707849 txs=0 appHash=A9E64B435B8DDD16C3BA9E043313865FC1BE552E362DB34180FC53A5858BD9E7
I[2020-02-04|08:58:00.508] Starting application REST service (chain-id: "euler-5")... module=rest-server
I[2020-02-04|08:58:00.508] Starting RPC HTTP server on [::]:1317 module=rest-server
I[2020-02-04|08:58:00.511] starting ABCI with Tendermint module=main
I[2020-02-04|08:58:00.858] Loading mem state module=main
I[2020-02-04|09:01:16.058] App loaded module=main time=3m15.199363809s
I[2020-02-04|09:01:23.583] Executed block module=state height=707850 validTxs=0 invalidTxs=0
I[2020-02-04|09:01:23.612] Committed state module=state height=707850 txs=0 appHash=A9E64B435B8DDD16C3BA9E043313865FC1BE552E362DB34180FC53A5858BD9E7
I[2020-02-04|09:01:23.619] Rank: data for gpu preparation module=main time=35.567155ms
I[2020-02-04|09:01:23.729] Rank: gpu calculations module=main time=109.560933ms
I[2020-02-04|09:01:23.729] Rank constructing tree module=main time=579.712µs
I[2020-02-04|09:01:23.729] Rank calculated module=main time=145.906585ms links=614 cids=600 hash=0ede28bd3934061573b7471d7a9832f2e9fdc40982a55f98e03a4640a4dd0550
I[2020-02-04|09:01:23.751] Executed block module=state height=707851 validTxs=0 invalidTxs=0
I[2020-02-04|09:01:23.783] Committed state module=state height=707851 txs=0 appHash=A9E64B435B8DDD16C3BA9E043313865FC1BE552E362DB34180FC53A5858BD9E7
I[2020-02-04|09:01:23.809] Executed block module=state height=707852 validTxs=0 invalidTxs=0
I[2020-02-04|09:01:23.843] Committed state module=state height=707852 txs=0 appHash=A9E64B435B8DDD16C3BA9E043313865FC1BE552E362DB34180FC53A5858BD9E7
I[2020-02-04|09:01:23.881] Executed block module=state height=707853 validTxs=0 invalidTxs=0
I[2020-02-04|09:01:23.909] Committed state module=state height=707853 txs=0 appHash=A9E64B435B8DDD16C3BA9E043313865FC1BE552E362DB34180FC53A5858BD9E7
I[2020-02-04|09:01:23.929] Executed block module=state height=707854 validTxs=0 invalidTxs=0
I[2020-02-04|09:01:23.959] Committed state module=state height=707854 txs=0 appHash=A9E64B435B8DDD16C3BA9E043313865FC1BE552E362DB34180FC53A5858BD9E7
I[2020-02-04|09:01:24.104] Executed block module=state height=707855 validTxs=0 invalidTxs=0
I[2020-02-04|09:01:24.140] Committed state module=state height=707855 txs=0 appHash=C61F26787090CFB9D2F9FBD0170948054DE331EA815083EA0D8D864D79AC2EBB
I[2020-02-04|09:01:24.140] Rank: data for gpu preparation module=main time=36.743021ms
I[2020-02-04|09:01:24.142] Rank: gpu calculations module=main time=1.807806ms
I[2020-02-04|09:01:24.143] Rank constructing tree module=main time=690.501µs
I[2020-02-04|09:01:24.143] Rank calculated module=main time=39.4916ms links=614 cids=600 hash=0ede28bd3934061573b7471d7a9832f2e9fdc40982a55f98e03a4640a4dd0550
panic: Failed to process committed block (707856:F3E49A161E44B03BA62E4DE69B928C5CF46CBD0B09F5EF57B012C458607EE104): Wrong Block.Header.AppHash. Expected C61F26787090CFB9D2F9FBD0170948054DE331EA815083EA0D8D864D79AC2EBB, got A9E64B435B8DDD16C3BA9E043313865FC1BE552E362DB34180FC53A5858BD9E7
goroutine 4038106 [running]:
github.com/tendermint/tendermint/blockchain/v0.(*BlockchainReactor).poolRoutine(0xc000b1a4e0)
/root/go/pkg/mod/github.com/tendermint/tendermint@v0.32.8/blockchain/v0/reactor.go:344 +0x12ff
created by github.com/tendermint/tendermint/blockchain/v0.(*BlockchainReactor).OnStart
/root/go/pkg/mod/github.com/tendermint/tendermint@v0.32.8/blockchain/v0/reactor.go:118 +0x84 |
|
Got a similar bug whilst trying to sync a new node from scratch Node wasn't stopped at any point.
|
We upgraded links storage from our custom one to sdk's in ongoing euler-6 (branch 0.1.6) |
In approx. 20% cases when docker container being stopped, and then started back up wrong app hash being calculated, node crashing.
The text was updated successfully, but these errors were encountered: