Too many open file handles

[greg-szabo]

Summary of Bug

gaiad and gaiacli uses too many file handles

Steps to Reproduce

(This is on gaia-seed4, while running gaia-9001. Seed4 was underutilized.)

[root@gaia-seed4 ~]# lsof | grep ^gaiad | wc -l
30058
[root@gaia-seed4 ~]# lsof | grep ^gaiacli | wc -l
42630
[root@gaia-seed4 ~]# lsof | grep -e '^gaiad.*TCP.*CLOSE_WAIT' | wc -l
5282
[root@gaia-seed4 ~]# lsof | grep -e '^gaiacli.*TCP.*CLOSE_WAIT' | wc -l
17934
[root@gaia-seed4 ~]# lsof | grep '^gaiad.*TCP.*ESTABLISHED' | wc -l
20786
[root@gaia-seed4 ~]# lsof | grep '^gaiacli.*TCP.*ESTABLISHED' | wc -l
21042
#gaiad process
[root@gaia-seed4 ~]# lsof -p 27226 | wc -l
1584
#gaiacli process
[root@gaia-seed4 ~]# lsof -p 16667 | wc -l
2031

It seems we need better TCP handling, but feel free to discuss it more.
Unrelated similar report from eariler: #1394

---

For Admin Use

• Not duplicate issue
• Appropriate labels applied
• Appropriate contributors tagged
• Contributor assigned/self-assigned

[ebuchman]

Why is gaiacli running on the seed node?

[greg-szabo]

Because when we first introduced seed nodes (around gaia-6000), the Voyager team reached out and said that they need the gaiacli LCD so they can run Voyager on the network and the Explorer website.
Since gaiacli required a full node but it works on the seed nodes too, we didn't set up full nodes separately for them.

[ebuchman]

Cool - but why does it have so many open TCP connections ?!

[ackratos]

Cool - but why does it have so many open TCP connections ?!

we experienced similar issue on our chain, some validators each hold 26000+ CLOSE_WAIT tcp connection with two witness nodes.

Checking witness node logs, it seems block reactor bug caused the top 2 blocks synced are in in-consistent state (second block's last commit id cannot match first block's commit id). As we don't have debug level log, cannot tell about how block reactor comes in this situation and cannot make sure all connections are caused by this. (restarting problematic validators can resolve this issue temporally)

I[12-28|11:26:37.790] Dialing peer                                 module=p2p address=638eabc55ac2cd1ec92e9edeac1c6d412a4a5d3c@10.203.43.118:26656
I[12-28|11:26:37.796] Starting Peer                                module=p2p peer=0xb0de50 impl="Peer{MConn{10.203.43.118:26656} 20cb9667e5c4040c510d3cd795361f4b98969798 out}"
I[12-28|11:26:37.796] Starting MConnection                         module=p2p peer=0xb0de50 impl=MConn{10.203.43.118:26656}
I[12-28|11:26:37.796] Added peer                                   module=p2p peer="Peer{MConn{10.203.43.118:26656} 20cb9667e5c4040c510d3cd795361f4b98969798 out}"
E[12-28|11:26:37.936] Error in validation                          module=blockchain err="Invalid commit -- wrong block id: want FB99188E4993C0C02C854D05BDD0DC9468BBCE0E:1:CB98105DAA13 got 01FF4061F5F229D21ECDE836D1A2E1C7AE9346CB:1:4D1563CAD2BB"
E[12-28|11:26:37.936] Stopping peer for error                      module=p2p peer="Peer{MConn{10.203.43.118:26656} 20cb9667e5c4040c510d3cd795361f4b98969798 out}" err="BlockchainReactor validation error: Invalid commit -- wrong block id: want FB99188E4993C0C02C854D05BDD0DC9468BBCE0E:1:CB98105DAA13 got 01FF4061F5F229D21ECDE836D1A2E1C7AE9346CB:1:4D1563CAD2BB"
I[12-28|11:26:37.936] Stopping Peer                                module=p2p peer=0xb0de50 impl="Peer{MConn{10.203.43.118:26656} 20cb9667e5c4040c510d3cd795361f4b98969798 out}"
I[12-28|11:26:37.936] Stopping MConnection                         module=p2p peer=0xb0de50 impl=MConn{10.203.43.118:26656}
I[12-28|11:26:37.937] Reconnecting to peer                         module=p2p addr=20cb9667e5c4040c510d3cd795361f4b98969798@10.203.43.118:26656

[jackzampolin]

@ackratos I think that might be more of a tendermint issue! The linked issue (tendermint#2967) might have some more information for you!

[jackzampolin]

Going to go ahead and close this as solved by systemd file and linux settings. We also point people to this fix in the docs.