Error on syncing fullnode

[astudnev]

I got the following error on block ~ 16 million:
ERROR: Error during handshake: Error on replay: Wrong Block.Header.AppHash. Expected 2902E81E5C1C9A789471CAF607CDCC2747A97A6C3257C9FA42D361A8D5E01D8B, got 937156DCB2220A0B323B4CD7040C368765F86DC76399090579AA37DB945DB66E

bnc.log shows:

I[2019-10-01|20:47:14.399] Added Order module=dex order="NewOrderMsg{Sender: 34AB7C199991F2A892E1D27E072D0C77AA3601A4, Id: 34AB7C199991F2A892E1D27E072D0C77AA3601A4-230, Symbol: LTO-BDF_BNB}"
I[2019-10-01|20:47:14.399] replayed all tx. Starting match module=dex height=16770239
I[2019-10-01|20:47:14.399] Relaying block for order book module=dex height=16770240
I[2019-10-01|20:47:14.399] Added Order module=dex order="NewOrderMsg{Sender: 4792B6F9891BD0B34235DC0F177C14EA21D4D23D, Id: 4792B6F9891BD0B34235DC0F177C14EA21D4D23D-356, Symbol: PHB-2DF_BNB}"
I[2019-10-01|20:47:14.399] replayed all tx. Starting match module=dex height=16770240
I[2019-10-01|20:47:14.399] Relaying block for order book module=dex height=16770241
I[2019-10-01|20:47:14.399] replayed all tx. Starting match module=dex height=16770241
I[2019-10-01|20:47:14.399] No new orders for any pair, give up matching module=dexkeeper
I[2019-10-01|20:47:14.399] No order comes in for the block module=dexkeeper
I[2019-10-01|20:47:14.399] Relaying block for order book module=dex height=16770242
I[2019-10-01|20:47:14.399] replayed all tx. Starting match module=dex height=16770242
I[2019-10-01|20:47:14.399] No new orders for any pair, give up matching module=dexkeeper
I[2019-10-01|20:47:14.399] No order comes in for the block module=dexkeeper
I[2019-10-01|20:47:14.399] Relaying block for order book module=dex height=16770243
I[2019-10-01|20:47:14.399] does not recover invalid tx module=dex txHash=54D51EFAD447F13CFEF76695D1CF48B39B7481B729836F025A27234155A84036 err="{"codespace":1,"code":1,"abci_code":65537,"message":"recovered: open /home/studnev/.bnbchaind/data/application.db/1349323.ldb: too many open files\nstack:\ngoroutine 39571 [running]:\nruntime/debug.Stack(0xc0014265c0, 0x109c540, 0xc00421c000)\n\t/usr/local/go/src/runtime/debug/stack.go:24 +0x9d\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).RunTx.func1(0xc001427aa8)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/baseapp/baseapp.go:821 +0x6e\npanic(0x109c540, 0xc00421c000)\n\t/usr/local/go/src/runtime/panic.go:522 +0x1b5\ngithub.com/binance-chain/node/vendor/github.com/tendermint/tendermint/libs/db.(*GoLevelDB).Get(0xc0004c4358, 0xc00af8e0c0, 0x29, 0x30, 0x29, 0xc00af8e0c0, 0x8)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/tendermint/libs/db/go_level_db.go:58 +0x137\ngithub.com/binance-chain/node/vendor/github.com/tendermint/tendermint/libs/db.(*prefixDB).Get(0xc000bda660, 0xc00af8e090, 0x21, 0x21, 0x0, 0x0, 0x0)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/tendermint/libs/db/prefix_db.go:60 +0x162\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*nodeDB).GetNode(0xc000b99810, 0xc006b1c120, 0x20, 0x20, 0x0)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/nodedb.go:80 +0x29e\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*Node).getRightNode(...)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/node.go:363\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*Node).get(0xc009cb0a50, 0xc008b8c320, 0xc0126a9ca0, 0x1c, 0x20, 0x1c, 0xffffffffffffffff, 0xc0014268f0, 0x65)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/node.go:176 +0x319\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*Node).get(0xc009cb0370, 0xc008b8c320, 0xc0126a9ca0, 0x1c, 0x20, 0x1c, 0x1, 0xc001426950, 0x65)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/node.go:174 +0x1aa\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*Node).get(0xc009cb02c0, 0xc008b8c320, 0xc0126a9ca0, 0x1c, 0x20, 0x1c, 0xffffffffffffffff, 0xc0014269b0, 0x65)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/node.go:177 +0x293\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*Node).get(0xc009cb0210, 0xc008b8c320, 0xc0126a9ca0, 0x1c, 0x20, 0x1c, 0x1, 0xc001426a10, 0x66)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/node.go:174 +0x1aa\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*Node).get(0xc012644dc0, 0xc008b8c320, 0xc0126a9ca0, 0x1c, 0x20, 0x1c, 0x1, 0xc001426a70, 0x66)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/node.go:177 +0x293\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*Node).get(0xc012644d10, 0xc008b8c320, 0xc0126a9ca0, 0x1c, 0x20, 0x1c, 0xffffffffffffffff, 0xc001426ad0, 0xc001426b90)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/node.go:177 +0x293\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*Node).get(0xc00853f1e0, 0xc008b8c320, 0xc0126a9ca0, 0x1c, 0x20, 0x1c, 0xffffffffffffffff, 0xc001426b30, 0xc001426bf0)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/node.go:174 +0x1aa\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*Node).get(0xc00606cbb0, 0xc008b8c320, 0xc0126a9ca0, 0x1c, 0x20, 0x1c, 0xffffffffffffffff, 0xc001426b90, 0xc001426c50)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/node.go:174 +0x1aa\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*Node).get(0xc00606c9a0, 0xc008b8c320, 0xc0126a9ca0, 0x1c, 0x20, 0x1c, 0x1, 0xc001426bf0, 0xc001426cb0)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/node.go:174 +0x1aa\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*Node).get(0xc00606c8f0, 0xc008b8c320, 0xc0126a9ca0, 0x1c, 0x20, 0x1c, 0xffffffffffffffff, 0xc001426c50, 0xc001426d10)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/node.go:177 +0x293\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*Node).get(0xc00817cdc0, 0xc008b8c320, 0xc0126a9ca0, 0x1c, 0x20, 0x1c, 0x1, 0xc001426cb0, 0xc001426d70)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/node.go:174 +0x1aa\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*Node).get(0xc00c5e7ad0, 0xc008b8c320, 0xc0126a9ca0, 0x1c, 0x20, 0x1c, 0xffffffffffffffff, 0xc001426d10, 0xc001426dd0)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/node.go:177 +0x293\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*Node).get(0xc00c5e78c0, 0xc008b8c320, 0xc0126a9ca0, 0x1c, 0x20, 0x1c, 0x1, 0x7fe6ed37c008, 0x41a309)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/node.go:174 +0x1aa\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*Node).get(0xc00bfd7760, 0xc008b8c320, 0xc0126a9ca0, 0x1c, 0x20, 0x1c, 0x1, 0x10000000043b5f1, 0x0)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/node.go:177 +0x293\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*Node).get(0xc00ce57340, 0xc008b8c320, 0xc0126a9ca0, 0x1c, 0x20, 0x1c, 0x1, 0xc014429690, 0xc001426e88)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/node.go:177 +0x293\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*Node).get(0xc012685080, 0xc008b8c320, 0xc0126a9ca0, 0x1c, 0x20, 0x20, 0x103a880, 0xc0126a9ca0, 0xc0126a9ca0)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/node.go:177 +0x293\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*ImmutableTree).Get(...)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/immutable_tree.go:98\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/store.(*IavlStore).Get(0xc002427760, 0xc0126a9ca0, 0x1c, 0x20, 0x1c, 0xc0126a9ca0, 0x8)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/store/iavlstore.go:153 +0x86\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/auth.(*accountStoreCache).GetAccount(0xc001c7a820, 0xc0126a9960, 0x14, 0x14, 0x0, 0x0)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/auth/mapper.go:236 +0x13d\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/auth.(*accountCache).getAccountFromCache(0xc005dda580, 0xc0126a9960, 0x14, 0x14, 0x1026560, 0xc001427060)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/auth/mapper.go:359 +0x14d\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/auth.(*accountCache).GetAccount(0xc005dda580, 0xc0126a9960, 0x14, 0x14, 0x0, 0x0)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/auth/mapper.go:299 +0x49\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/auth.(*accountCache).getAccountFromCache(0xc005dda800, 0xc0126a9960, 0x14, 0x14, 0x113a200, 0x15001a0)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/auth/mapper.go:359 +0x14d\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/auth.(*accountCache).GetAccount(0xc005dda800, 0xc0126a9960, 0x14, 0x14, 0xc005dda800, 0x14b49b8)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/auth/mapper.go:299 +0x49\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/auth.AccountKeeper.GetAccount(0x14e86c0, 0xc000076db0, 0x136fe68, 0xc0004cfab0, 0x14f7580, 0xc017eab260, 0xc005dda600, 0x10, 0xc0126a9960, 0x14, ...)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/auth/mapper.go:87 +0xcb\ngithub.com/binance-chain/node/common/tx.processAccount(0x14f7580, 0xc017eab260, 0xc005dda600, 0x10, 0x14e86c0, 0xc000076db0, 0x136fe68, 0xc0004cfab0, 0xc0126a9960, 0x14, ...)\n\t/home/suyu/go/src/github.com/binance-chain/node/common/tx/ante.go:233 +0xc2\ngithub.com/binance-chain/node/common/tx.NewAnteHandler.func1(0x14f7580, 0xc017eab260, 0xc005dda600, 0x10, 0x14d9a20, 0xc009872840, 0x1026505, 0xc014429670, 0x14f7580, 0xc017eab260, ...)\n\t/home/suyu/go/src/github.com/binance-chain/node/common/tx/ante.go:151 +0x599\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).RunTx(0xc0002ede00, 0x11fb905, 0x14d9a20, 0xc009872840, 0xc00a6c7840, 0x40, 0x0, 0x0, 0x0, 0x0, ...)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/baseapp/baseapp.go:834 +0x6e3\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).DeliverTx(0xc0002ede00, 0xc004576ff0, 0xe1, 0xe1, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/baseapp/baseapp.go:657 +0x270\ngithub.com/binance-chain/node/app.(*BinanceChain).DeliverTx(0xc0004d8500, 0xc004576ff0, 0xe1, 0xe1, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)\n\t/home/suyu/go/src/github.com/binance-chain/node/app/app.go:440 +0x9b\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/server/concurrent.(*asyncLocalClient).deliverTxWorker.func1(0xc000b10a50, 0xc001427fa8)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/server/concurrent/async_local_client.go:142 +0x1b6\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/server/concurrent.(*asyncLocalClient).deliverTxWorker(0xc000b10a50)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/server/concurrent/async_local_client.go:151 +0x8c\ncreated by github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/server/concurrent.(*asyncLocalClient).OnStart\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/server/concurrent/async_local_client.go:93 +0x64\n"}" height=16770243
I[2019-10-01|20:47:14.399] replayed all tx. Starting match module=dex height=16770243
I[2019-10-01|20:47:14.399] No new orders for any pair, give up matching module=dexkeeper
I[2019-10-01|20:47:14.399] No order comes in for the block module=dexkeeper
I[2019-10-01|20:47:14.638] Located day to load breathe block height module=dexkeeper epochDay="\ufffdF\u0000\u0000\u0000\u0000\u0000\u0000"
I[2019-10-01|20:47:14.638] Loaded breathe block height module=dexkeeper height=16689772
I[2019-10-01|20:47:14.638] get last breathe block height module=main height=16689772
I[2019-10-01|20:47:14.645] reload latest snapshot module=statesync height=16689772
I[2019-10-01|20:47:14.836] ABCI Handshake App Info module=consensus height=16770243 hash=2902E81E5C1C9A789471CAF607CDCC2747A97A6C3257C9FA42D361A8D5E01D8B software-version= protocol-version=0
I[2019-10-01|20:47:14.838] ABCI Replay Blocks module=consensus appHeight=16770243 storeHeight=16770244 stateHeight=16770243
I[2019-10-01|20:47:14.838] Replay last block using real app module=consensus

[astudnev]

version 0.6.1. Ubuntu OS

[astudnev]

Problem is that even after extending ulimit it does not recover

[chainwhisper]

@astudnev
How much limit did you set?

What's the new error after you restart?

[astudnev]

I set ulimit 65535 and after restart getting the same error as i posted

[chainwhisper]

@astudnev

Hello, can you please share the output for ulimit -a

[unclezoro]

@astudnev what is your log now after you set ulimit 65535 , can you post your app.toml as well?

[astudnev]

ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 256696
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 65535
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 256696
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

[astudnev]

./linux/bnbchaind --home ~/.bnbchaind start
ERROR: Error during handshake: Error on replay: Wrong Block.Header.AppHash. Expected 2902E81E5C1C9A789471CAF607CDCC2747A97A6C3257C9FA42D361A8D5E01D8B, got 937156DCB2220A0B323B4CD7040C368765F86DC76399090579AA37DB945DB66E

[astudnev]

bnc.log tail

I[2019-10-03|18:53:48.833] replayed all tx. Starting match module=dex height=16770236
I[2019-10-03|18:53:48.833] No new orders for any pair, give up matching module=dexkeeper
I[2019-10-03|18:53:48.833] No order comes in for the block module=dexkeeper
I[2019-10-03|18:53:48.833] Relaying block for order book module=dex height=16770237
I[2019-10-03|18:53:48.833] replayed all tx. Starting match module=dex height=16770237
I[2019-10-03|18:53:48.833] No new orders for any pair, give up matching module=dexkeeper
I[2019-10-03|18:53:48.833] No order comes in for the block module=dexkeeper
I[2019-10-03|18:53:48.833] Relaying block for order book module=dex height=16770238
I[2019-10-03|18:53:48.833] replayed all tx. Starting match module=dex height=16770238
I[2019-10-03|18:53:48.833] No new orders for any pair, give up matching module=dexkeeper
I[2019-10-03|18:53:48.833] No order comes in for the block module=dexkeeper
I[2019-10-03|18:53:48.833] Relaying block for order book module=dex height=16770239
I[2019-10-03|18:53:48.834] Added Order module=dex order="NewOrderMsg{Sender: 34AB7C199991F2A892E1D27E072D0C77AA3601A4, Id: 34AB7C199991F2A892E1D27E072D0C77AA3601A4-230, Symbol: LTO-BDF_BNB}"
I[2019-10-03|18:53:48.834] replayed all tx. Starting match module=dex height=16770239
I[2019-10-03|18:53:48.834] Relaying block for order book module=dex height=16770240
I[2019-10-03|18:53:48.834] Added Order module=dex order="NewOrderMsg{Sender: 4792B6F9891BD0B34235DC0F177C14EA21D4D23D, Id: 4792B6F9891BD0B34235DC0F177C14EA21D4D23D-356, Symbol: PHB-2DF_BNB}"
I[2019-10-03|18:53:48.834] replayed all tx. Starting match module=dex height=16770240
I[2019-10-03|18:53:48.834] Relaying block for order book module=dex height=16770241
I[2019-10-03|18:53:48.834] replayed all tx. Starting match module=dex height=16770241
I[2019-10-03|18:53:48.834] No new orders for any pair, give up matching module=dexkeeper
I[2019-10-03|18:53:48.834] No order comes in for the block module=dexkeeper
I[2019-10-03|18:53:48.834] Relaying block for order book module=dex height=16770242
I[2019-10-03|18:53:48.834] replayed all tx. Starting match module=dex height=16770242
I[2019-10-03|18:53:48.834] No new orders for any pair, give up matching module=dexkeeper
I[2019-10-03|18:53:48.834] No order comes in for the block module=dexkeeper
I[2019-10-03|18:53:48.834] Relaying block for order book module=dex height=16770243
I[2019-10-03|18:53:48.834] does not recover invalid tx module=dex txHash=54D51EFAD447F13CFEF76695D1CF48B39B7481B729836F025A27234155A84036 err="{"codespace":1,"code":1,"abci_code":65537,"message":"recovered: open /home/studnev/.bnbchaind/data/application.db/1349323.ldb: too many open files\nstack:\ngoroutine 39571 [running]:\nruntime/debug.Stack(0xc0014265c0, 0x109c540, 0xc00421c000)\n\t/usr/local/go/src/runtime/debug/stack.go:24 +0x9d\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).RunTx.func1(0xc001427aa8)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/baseapp/baseapp.go:821 +0x6e\npanic(0x109c540, 0xc00421c000)\n\t/usr/local/go/src/runtime/panic.go:522 +0x1b5\ngithub.com/binance-chain/node/vendor/github.com/tendermint/tendermint/libs/db.(*GoLevelDB).Get(0xc0004c4358, 0xc00af8e0c0, 0x29, 0x30, 0x29, 0xc00af8e0c0, 0x8)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/tendermint/libs/db/go_level_db.go:58 +0x137\ngithub.com/binance-chain/node/vendor/github.com/tendermint/tendermint/libs/db.(*prefixDB).Get(0xc000bda660, 0xc00af8e090, 0x21, 0x21, 0x0, 0x0, 0x0)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/tendermint/libs/db/prefix_db.go:60 +0x162\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*nodeDB).GetNode(0xc000b99810, 0xc006b1c120, 0x20, 0x20, 0x0)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/nodedb.go:80 +0x29e\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*Node).getRightNode(...)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/node.go:363\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*Node).get(0xc009cb0a50, 0xc008b8c320, 0xc0126a9ca0, 0x1c, 0x20, 0x1c, 0xffffffffffffffff, 0xc0014268f0, 0x65)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/node.go:176 +0x319\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*Node).get(0xc009cb0370, 0xc008b8c320, 0xc0126a9ca0, 0x1c, 0x20, 0x1c, 0x1, 0xc001426950, 0x65)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/node.go:174 +0x1aa\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*Node).get(0xc009cb02c0, 0xc008b8c320, 0xc0126a9ca0, 0x1c, 0x20, 0x1c, 0xffffffffffffffff, 0xc0014269b0, 0x65)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/node.go:177 +0x293\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*Node).get(0xc009cb0210, 0xc008b8c320, 0xc0126a9ca0, 0x1c, 0x20, 0x1c, 0x1, 0xc001426a10, 0x66)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/node.go:174 +0x1aa\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*Node).get(0xc012644dc0, 0xc008b8c320, 0xc0126a9ca0, 0x1c, 0x20, 0x1c, 0x1, 0xc001426a70, 0x66)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/node.go:177 +0x293\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*Node).get(0xc012644d10, 0xc008b8c320, 0xc0126a9ca0, 0x1c, 0x20, 0x1c, 0xffffffffffffffff, 0xc001426ad0, 0xc001426b90)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/node.go:177 +0x293\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*Node).get(0xc00853f1e0, 0xc008b8c320, 0xc0126a9ca0, 0x1c, 0x20, 0x1c, 0xffffffffffffffff, 0xc001426b30, 0xc001426bf0)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/node.go:174 +0x1aa\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*Node).get(0xc00606cbb0, 0xc008b8c320, 0xc0126a9ca0, 0x1c, 0x20, 0x1c, 0xffffffffffffffff, 0xc001426b90, 0xc001426c50)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/node.go:174 +0x1aa\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*Node).get(0xc00606c9a0, 0xc008b8c320, 0xc0126a9ca0, 0x1c, 0x20, 0x1c, 0x1, 0xc001426bf0, 0xc001426cb0)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/node.go:174 +0x1aa\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*Node).get(0xc00606c8f0, 0xc008b8c320, 0xc0126a9ca0, 0x1c, 0x20, 0x1c, 0xffffffffffffffff, 0xc001426c50, 0xc001426d10)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/node.go:177 +0x293\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*Node).get(0xc00817cdc0, 0xc008b8c320, 0xc0126a9ca0, 0x1c, 0x20, 0x1c, 0x1, 0xc001426cb0, 0xc001426d70)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/node.go:174 +0x1aa\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*Node).get(0xc00c5e7ad0, 0xc008b8c320, 0xc0126a9ca0, 0x1c, 0x20, 0x1c, 0xffffffffffffffff, 0xc001426d10, 0xc001426dd0)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/node.go:177 +0x293\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*Node).get(0xc00c5e78c0, 0xc008b8c320, 0xc0126a9ca0, 0x1c, 0x20, 0x1c, 0x1, 0x7fe6ed37c008, 0x41a309)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/node.go:174 +0x1aa\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*Node).get(0xc00bfd7760, 0xc008b8c320, 0xc0126a9ca0, 0x1c, 0x20, 0x1c, 0x1, 0x10000000043b5f1, 0x0)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/node.go:177 +0x293\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*Node).get(0xc00ce57340, 0xc008b8c320, 0xc0126a9ca0, 0x1c, 0x20, 0x1c, 0x1, 0xc014429690, 0xc001426e88)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/node.go:177 +0x293\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*Node).get(0xc012685080, 0xc008b8c320, 0xc0126a9ca0, 0x1c, 0x20, 0x20, 0x103a880, 0xc0126a9ca0, 0xc0126a9ca0)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/node.go:177 +0x293\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*ImmutableTree).Get(...)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/immutable_tree.go:98\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/store.(*IavlStore).Get(0xc002427760, 0xc0126a9ca0, 0x1c, 0x20, 0x1c, 0xc0126a9ca0, 0x8)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/store/iavlstore.go:153 +0x86\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/auth.(*accountStoreCache).GetAccount(0xc001c7a820, 0xc0126a9960, 0x14, 0x14, 0x0, 0x0)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/auth/mapper.go:236 +0x13d\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/auth.(*accountCache).getAccountFromCache(0xc005dda580, 0xc0126a9960, 0x14, 0x14, 0x1026560, 0xc001427060)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/auth/mapper.go:359 +0x14d\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/auth.(*accountCache).GetAccount(0xc005dda580, 0xc0126a9960, 0x14, 0x14, 0x0, 0x0)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/auth/mapper.go:299 +0x49\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/auth.(*accountCache).getAccountFromCache(0xc005dda800, 0xc0126a9960, 0x14, 0x14, 0x113a200, 0x15001a0)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/auth/mapper.go:359 +0x14d\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/auth.(*accountCache).GetAccount(0xc005dda800, 0xc0126a9960, 0x14, 0x14, 0xc005dda800, 0x14b49b8)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/auth/mapper.go:299 +0x49\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/auth.AccountKeeper.GetAccount(0x14e86c0, 0xc000076db0, 0x136fe68, 0xc0004cfab0, 0x14f7580, 0xc017eab260, 0xc005dda600, 0x10, 0xc0126a9960, 0x14, ...)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/auth/mapper.go:87 +0xcb\ngithub.com/binance-chain/node/common/tx.processAccount(0x14f7580, 0xc017eab260, 0xc005dda600, 0x10, 0x14e86c0, 0xc000076db0, 0x136fe68, 0xc0004cfab0, 0xc0126a9960, 0x14, ...)\n\t/home/suyu/go/src/github.com/binance-chain/node/common/tx/ante.go:233 +0xc2\ngithub.com/binance-chain/node/common/tx.NewAnteHandler.func1(0x14f7580, 0xc017eab260, 0xc005dda600, 0x10, 0x14d9a20, 0xc009872840, 0x1026505, 0xc014429670, 0x14f7580, 0xc017eab260, ...)\n\t/home/suyu/go/src/github.com/binance-chain/node/common/tx/ante.go:151 +0x599\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).RunTx(0xc0002ede00, 0x11fb905, 0x14d9a20, 0xc009872840, 0xc00a6c7840, 0x40, 0x0, 0x0, 0x0, 0x0, ...)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/baseapp/baseapp.go:834 +0x6e3\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).DeliverTx(0xc0002ede00, 0xc004576ff0, 0xe1, 0xe1, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/baseapp/baseapp.go:657 +0x270\ngithub.com/binance-chain/node/app.(*BinanceChain).DeliverTx(0xc0004d8500, 0xc004576ff0, 0xe1, 0xe1, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)\n\t/home/suyu/go/src/github.com/binance-chain/node/app/app.go:440 +0x9b\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/server/concurrent.(*asyncLocalClient).deliverTxWorker.func1(0xc000b10a50, 0xc001427fa8)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/server/concurrent/async_local_client.go:142 +0x1b6\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/server/concurrent.(*asyncLocalClient).deliverTxWorker(0xc000b10a50)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/server/concurrent/async_local_client.go:151 +0x8c\ncreated by github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/server/concurrent.(*asyncLocalClient).OnStart\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/server/concurrent/async_local_client.go:93 +0x64\n"}" height=16770243
I[2019-10-03|18:53:48.834] replayed all tx. Starting match module=dex height=16770243
I[2019-10-03|18:53:48.834] No new orders for any pair, give up matching module=dexkeeper
I[2019-10-03|18:53:48.834] No order comes in for the block module=dexkeeper
I[2019-10-03|18:53:49.070] Located day to load breathe block height module=dexkeeper epochDay="\ufffdF\u0000\u0000\u0000\u0000\u0000\u0000"
I[2019-10-03|18:53:49.070] Loaded breathe block height module=dexkeeper height=16689772
I[2019-10-03|18:53:49.070] get last breathe block height module=main height=16689772
I[2019-10-03|18:53:49.077] reload latest snapshot module=statesync height=16689772
I[2019-10-03|18:53:49.266] ABCI Handshake App Info module=consensus height=16770243 hash=2902E81E5C1C9A789471CAF607CDCC2747A97A6C3257C9FA42D361A8D5E01D8B software-version= protocol-version=0
I[2019-10-03|18:53:49.273] ABCI Replay Blocks module=consensus appHeight=16770243 storeHeight=16770244 stateHeight=16770243
I[2019-10-03|18:53:49.273] Replay last block using real app module=consensus

[chainwhisper]

it seems you are on mainnet. Which version are you using?

[astudnev]

0.6.1

[chainwhisper]

0.6.1

can you please share your app.toml file please?

[astudnev]

cat .bnbchaind/config/app.toml 
# This is a TOML config file.
# For more information, see https://github.com/toml-lang/toml

[base]
# Interval blocks of breathe block, if breatheBlockInterval is 0, breathe block will be created every day.
breatheBlockInterval = 0
# Size of account cache
accountCacheSize = 30000
# Size of signature cache
signatureCacheSize = 30000
# Running mode when start up, 0: Normal, 1: TransferOnly, 2: RecoverOnly
startMode = 0
# Concurrency of OrderKeeper, should be power of 2
orderKeeperConcurrency = 2
# Days count back for breathe block
breatheBlockDaysCountBack = 7

[upgrade]
# Block height of BEP6 upgrade
BEP6Height = 20300000
# Block height of BEP9 upgrade
BEP9Height = 20300000
# Block height of BEP10 upgrade
BEP10Height = 20300000
# Block height of BEP19 upgrade
BEP19Height = 20300000
# Block height of BEP12 upgrade
BEP12Height = 25120000

[addr]
# Bech32PrefixAccAddr defines the Bech32 prefix of an account's address
bech32PrefixAccAddr = "bnb"
# Bech32PrefixAccPub defines the Bech32 prefix of an account's public key
bech32PrefixAccPub = "bnbp"
# Bech32PrefixValAddr defines the Bech32 prefix of a validator's operator address
bech32PrefixValAddr = "bva"
# Bech32PrefixValPub defines the Bech32 prefix of a validator's operator public key
bech32PrefixValPub = "bvap"
# Bech32PrefixConsAddr defines the Bech32 prefix of a consensus node address
bech32PrefixConsAddr = "bca"
# Bech32PrefixConsPub defines the Bech32 prefix of a consensus node public key
bech32PrefixConsPub = "bcap"

##### publication related configurations #####
[publication]
# configurations ends with Kafka can be a semi-colon separated host-port list
# Whether we want publish market data (this includes trades and order)
publishOrderUpdates = true
orderUpdatesTopic = "orders"
orderUpdatesKafka = "127.0.0.1:9092"

# Whether we want publish account balance to notify browser db indexer persist latest account balance change
publishAccountBalance = true
accountBalanceTopic = "accounts"
accountBalanceKafka = "127.0.0.1:9092"

# Whether we want publish order book changes
publishOrderBook = true
orderBookTopic = "orders"
orderBookKafka = "127.0.0.1:9092"

# Whether we want publish block fee changes
publishBlockFee = true
blockFeeTopic = "accounts"
blockFeeKafka = "127.0.0.1:9092"

# Whether we want publish transfers
publishTransfer = true
transferTopic = "transfers"
transferKafka = "127.0.0.1:9092"

# Global setting
publicationChannelSize = "10000"
publishKafka = false
publishLocal = true



# max size in megabytes of marketdata json file before rotate
localMaxSize = 1024
# max days of marketdata json files to keep before deleted
localMaxAge = 7000

[log]

# Write logs to console instead of file
logToConsole = false

## The below parameters take effect only when logToConsole is false
# Log file root, if not set, use home path
logFileRoot = ""
# Log file path relative to log file root path
logFilePath = "bnc.log"
# Number of logs keep in memory before writing to file
logBuffSize = 10000`

[unclezoro]

@astudnev Thks for report the issue. I think it is a potential bug for binance chain. The reason is that at first the machine that run bnbchaind do not set large enough open files, and cause one Transaction failed(which is supposed to be successed). As I see, your fullnode have synced 16770243 blocks while the chain is in 43159598, you can clean the data and use state-sync(will help you quick catch up). Please refer to https://docs.binance.org/fullnode.html. As you have configed 65535 max open files, it won't happen again.

[chainwhisper]

I think we should backup data folder first, then use https://github.com/binance-chain/node-binary/tree/master/tools/recover tool to rollback to height 16770242. then restart

[astudnev]

OK, thank you, i already synced again.

[chainwhisper]

OK, thank you, i already synced again.

Great!

Did you re-sync from the start?

[astudnev]

OK, thank you, i already synced again.

Great!

Did you re-sync from the start?

yes

[astudnev]

I also tried recover tool, it worked fine in this situation