bnbcli does not close the open file handle

[singpenguin]

The bnbcli will throw "too many files".

goroutine 1 [running]:
github.com/binance-chain/node/vendor/github.com/tendermint/tendermint/libs/db.(*goLevelDBIterator).assertNoError(0xc003ce0b90)
/bnchome/src/github.com/binance-chain/node/vendor/github.com/tendermint/tendermint/libs/db/go_level_db.go:332 +0x60
github.com/binance-chain/node/vendor/github.com/tendermint/tendermint/libs/db.(*goLevelDBIterator).Valid(0xc003ce0b90, 0x114c238)
/bnchome/src/github.com/binance-chain/node/vendor/github.com/tendermint/tendermint/libs/db/go_level_db.go:267 +0x3e
github.com/binance-chain/node/vendor/github.com/tendermint/tendermint/lite.(*DBProvider).LatestFullCommit(0xc000076e00, 0x7ffd8a41ae5c, 0x14, 0x9ea8d9, 0x9ea8d9, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/bnchome/src/github.com/binance-chain/node/vendor/github.com/tendermint/tendermint/lite/dbprovider.go:117 +0x38b
github.com/binance-chain/node/vendor/github.com/tendermint/tendermint/lite.(*multiProvider).LatestFullCommit(0xc00223ec00, 0x7ffd8a41ae5c, 0x14, 0x9ea8d9, 0x9ea8d9, 0xc000014c80, 0x0, 0x0, 0x0, 0x0, ...)
/bnchome/src/github.com/binance-chain/node/vendor/github.com/tendermint/tendermint/lite/multiprovider.go:51 +0xe1
github.com/binance-chain/node/vendor/github.com/tendermint/tendermint/lite.(*DynamicVerifier).Verify(0xc003ce0a00, 0xc0043874a0, 0xc000014c80, 0x0, 0x0)
/bnchome/src/github.com/binance-chain/node/vendor/github.com/tendermint/tendermint/lite/dynamic_verifier.go:95 +0x110
github.com/binance-chain/node/vendor/github.com/tendermint/tendermint/lite/proxy.GetCertifiedCommit(0x9ea8d9, 0x1154d00, 0xc0009c9400, 0x11413a0, 0xc003ce0a00, 0xd792a0, 0xc0000a9080, 0xd792a0, 0xc0000a9080)
/bnchome/src/github.com/binance-chain/node/vendor/github.com/tendermint/tendermint/lite/proxy/query.go:139 +0x124
github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/client/context.CLIContext.Verify(0xc00019aee0, 0x0, 0x1154d00, 0xc0009c9400, 0x113e580, 0xc00000e018, 0x0, 0x7ffd8a41ae78, 0x15, 0x0, ...)
/bnchome/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/client/context/query.go:187 +0x66
github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/client/tx.ValidateTxResult(0xc00019aee0, 0x0, 0x1154d00, 0xc0009c9400, 0x113e580, 0xc00000e018, 0x0, 0x7ffd8a41ae78, 0x15, 0x0, ...)
/bnchome/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/client/tx/query.go:91 +0x77
github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/client/tx.queryTx(0xc00019aee0, 0xc00019aee0, 0x0, 0x1154d00, 0xc0009c9400, 0x113e580, 0xc00000e018, 0x0, 0x7ffd8a41ae78, 0x15, ...)
/bnchome/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/client/tx/query.go:72 +0x43b
github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/client/tx.QueryTxCmd.func1(0xc0009406c0, 0xc000341a40, 0x1, 0x7, 0x0, 0x0)
/bnchome/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/client/tx/query.go:36 +0x199
github.com/binance-chain/node/vendor/github.com/spf13/cobra.(*Command).execute(0xc0009406c0, 0xc0003418f0, 0x7, 0x7, 0xc0009406c0, 0xc0003418f0)
/bnchome/src/github.com/binance-chain/node/vendor/github.com/spf13/cobra/command.go:698 +0x47a
github.com/binance-chain/node/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0x1a68d80, 0x24, 0xc0004740ac, 0xf2276b)
/bnchome/src/github.com/binance-chain/node/vendor/github.com/spf13/cobra/command.go:783 +0x2dc
github.com/binance-chain/node/vendor/github.com/spf13/cobra.(*Command).Execute(0x1a68d80, 0x1, 0xc00000ebf8)
/bnchome/src/github.com/binance-chain/node/vendor/github.com/spf13/cobra/command.go:736 +0x2b
github.com/binance-chain/node/vendor/github.com/tendermint/tendermint/libs/cli.Executor.Execute(0x1a68d80, 0x107cdf0, 0x2, 0xf116f5)
/bnchome/src/github.com/binance-chain/node/vendor/github.com/tendermint/tendermint/libs/cli/setup.go:89 +0x4e
main.main()
/bnchome/src/github.com/binance-chain/node/cmd/bnbcli/main.go:102 +0x5dd

[abelliumnt]

Could provide more information about how to reproduce this issue?

[ruslansalikhov]

Hi everybody. Just come here to open the same issue.

I tried to scan blockchain on own fullnode. But after gettings ~100 blocks I got error:

panic: open /root/.bnbchaind/data/application.db/319782.ldb: too many open files

goroutine 33475 [running]:
github.com/binance-chain/node/vendor/github.com/tendermint/tendermint/libs/db.(*GoLevelDB).Get(0xc0000143a8, 0xc02f996270, 0x29, 0x30, 0x29, 0xc02f996270, 0x8)
        /Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/tendermint/libs/db/go_level_db.go:58 +0x137
github.com/binance-chain/node/vendor/github.com/tendermint/tendermint/libs/db.(*prefixDB).Get(0xc000bef620, 0xc02f996240, 0x21, 0x21, 0x0, 0x0, 0x0)
        /Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/tendermint/libs/db/prefix_db.go:60 +0x162
github.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*nodeDB).GetNode(0xc0005140f0, 0xc0068cb560, 0x20, 0x20, 0x0)
        /Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/nodedb.go:76 +0x29e
github.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*Node).getLeftNode(...)
        /Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/node.go:354
github.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*MutableTree).recursiveSet(0xc000bef6e0, 0xc08f0a6a50, 0xc0062a5700, 0x1c, 0x20, 0xc07a55e360, 0x89, 0x90, 0x42c8ff, 0x8, ...)
        /Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/mutable_tree.go:121 +0xc97
github.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*MutableTree).recursiveSet(0xc000bef6e0, 0xc08f0a69a0, 0xc0062a5700, 0x1c, 0x20, 0xc07a55e360, 0x89, 0x90, 0x42c8ff, 0x8, ...)
        /Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/mutable_tree.go:126 +0xd58
github.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*MutableTree).recursiveSet(0xc000bef6e0, 0xc08d0ae420, 0xc0062a5700, 0x1c, 0x20, 0xc07a55e360, 0x89, 0x90, 0x42c8ff, 0x8, ...)
        /Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/mutable_tree.go:126 +0xd58
github.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*MutableTree).recursiveSet(0xc000bef6e0, 0xc08d0ae370, 0xc0062a5700, 0x1c, 0x20, 0xc07a55e360, 0x89, 0x90, 0x42c8ff, 0x8, ...)
        /Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/mutable_tree.go:126 +0xd58
github.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*MutableTree).recursiveSet(0xc000bef6e0, 0xc055284790, 0xc0062a5700, 0x1c, 0x20, 0xc07a55e360, 0x89, 0x90, 0xc058bde3c0, 0x30, ...)
        /Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/mutable_tree.go:126 +0xd58
github.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*MutableTree).recursiveSet(0xc000bef6e0, 0xc0df71f4a0, 0xc0062a5700, 0x1c, 0x20, 0xc07a55e360, 0x89, 0x90, 0x40beb9, 0xc00096c5a8, ...)
        /Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/mutable_tree.go:121 +0x868
github.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*MutableTree).set(0xc000bef6e0, 0xc0062a5700, 0x1c, 0x20, 0xc07a55e360, 0x89, 0x90, 0xc0dd6ae800, 0x109c380, 0xc00096c640, ...)
        /Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/mutable_tree.go:82 +0xab
github.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*MutableTree).Set(0xc000bef6e0, 0xc0062a5700, 0x1c, 0x20, 0xc07a55e360, 0x89, 0x90, 0xc0062a5700)
        /Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/mutable_tree.go:69 +0x7a
github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/store.(*IavlStore).Set(0xc02cad80c0, 0xc0062a5700, 0x1c, 0x20, 0xc07a55e360, 0x89, 0x90)
        /Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/store/iavlstore.go:126 +0x6d
github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/auth.(*accountStoreCache).SetAccount(0xc086b41b80, 0xc0062a56c0, 0x14, 0x20, 0x13fb880, 0xc1405a8b40)
        /Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/auth/mapper.go:253 +0x1f0
github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/auth.(*accountCache).Write(0xc1135cab00)
        /Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/auth/mapper.go:348 +0x26f
github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/baseapp.(*state).WriteAccountCache(...)
        /Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/baseapp/baseapp.go:286
github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).Commit(0xc0009cf950, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
        /Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/baseapp/baseapp.go:924 +0x187
github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/server/concurrent.(*asyncLocalClient).CommitSync(0xc142aec370, 0x0, 0x0, 0x0)
        /Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/server/concurrent/async_local_client.go:389 +0x1d7
github.com/binance-chain/node/vendor/github.com/tendermint/tendermint/proxy.(*appConnConsensus).CommitSync(0xc07d98eef0, 0x0, 0x0, 0x1)
        /Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/tendermint/proxy/app_conn.go:134 +0x33
github.com/binance-chain/node/vendor/github.com/tendermint/tendermint/state.(*BlockExecutor).Commit(0xc07a9bd420, 0xa, 0x0, 0xc015748e20, 0x6, 0xc01505fee0, 0x14, 0x8fcaee, 0x749ca, 0xc0aff0a9c0, ...)
        /Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/tendermint/state/execution.go:204 +0x236
github.com/binance-chain/node/vendor/github.com/tendermint/tendermint/state.(*BlockExecutor).ApplyBlock(0xc07a9bd420, 0xa, 0x0, 0xc015748e20, 0x6, 0xc01505fee0, 0x14, 0x8fcaee, 0x749ca, 0xc0aff0a9c0, ...)
        /Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/tendermint/state/execution.go:159 +0x76c
github.com/binance-chain/node/vendor/github.com/tendermint/tendermint/blockchain.(*BlockchainReactor).poolRoutine(0xc013972340)
        /Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/tendermint/blockchain/reactor.go:372 +0xbcb
created by github.com/binance-chain/node/vendor/github.com/tendermint/tendermint/blockchain.(*BlockchainReactor).OnStart
        /Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/tendermint/blockchain/reactor.go:117 +0x84

Steps to reproduce

1. Run full node
2. Request 100-2000 blocks via rpc (http/websocket - not matters)
3. Crash

Example how to request:

for i in {1..10000}; do echo $i; res=$(curl -s http://127.0.0.1:27147/block?height=$i | jq .result); if [[ $res == "" ]]; then echo ERROR; break; fi; done

[abelliumnt]

Can you try again with increased system setting about maximum open files? Usually, by default, its value is 1024. You can try with 10240.

[ruslansalikhov]

@HaoyangLiu

Yep. I will try (but I think it will increase number of blocks which I can get before crash). I will try soon - when I will restore my node.

I'm getting now another issue (after restart crashed node)

panic: Failed to process committed block (9491009:0B8E075E4988A46EB50E229CD2638C5D7A581BD6C9FFB66786B04BAF0EB1871A): Wrong Block.Header.AppHash.  Expected 88900FE61249CF354F3903A083129E6D28585242F93948DCC3476BE88047E0D0, got C2DD3ED4F64521362784D4AF7CEE547ADC97D02B29E9692133D88B61B3933940

goroutine 2040 [running]:
github.com/binance-chain/node/vendor/github.com/tendermint/tendermint/blockchain.(*BlockchainReactor).poolRoutine(0xc09e960ea0)
        /Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/tendermint/blockchain/reactor.go:375 +0x1ced
created by github.com/binance-chain/node/vendor/github.com/tendermint/tendermint/blockchain.(*BlockchainReactor).OnStart
        /Users/huangsuyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/tendermint/blockchain/reactor.go:117 +0x84

[ruslansalikhov]

Can you try again with increased system setting about maximum open files? Usually, by default, its value is 1024. You can try with 10240.

Thanks. Now it works fine. Processed >50K blocks already.

[abelliumnt]

If you encounter appHash conflict issue, please check your configuration in app.toml. For recommended configuration please refer to this page.

[singpenguin]

@HaoyangLiu I tried setting system's maximum open files to 65535, but the process's max open files is 1024,
eg. cat /proc/XXX/limits

Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size 0 unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 14727 14727 processes
Max open files 1024 4096 files
Max locked memory 16777216 16777216 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 14727 14727 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us

[singpenguin]

The way it appears is the same as ruslansalikhov

#51 (comment)

[abelliumnt]

Max open files 1024 4096 files

Maybe your setting was not correct. Please refer to this document