Gracefully disconnect from seed nodes

[cwgoes]

Presently, nodes running in seed mode drop all p2p connections after sending peers, but those peers treat the dropped connection as a normal failure, resulting in lots of errors of the form

I[07-16|21:08:59.284] Successful handshake with peer               module=p2p peer=178.128.206.155:26656 peerNodeInfo="NodeInfo{id: 5922bf29b48a18c2300b85cc53f424fce23927ab, moniker: adrian, network: gaia-7000 [listen 178.128.206.155:26656], version: 0.22.2 ([amino_version=0.10.1 p2p_version=0.5.0 consensus_version=v1/0.2.2 rpc_version=0.7.0/3 tx_index=on rpc_addr=tcp://0.0.0.0:26657])}"
I[07-16|21:08:59.284] Starting Peer                                module=p2p peer=178.128.206.155:26656 impl="Peer{MConn{178.128.206.155:26656} 5922bf29b48a18c2300b85cc53f424fce23927ab out}"
I[07-16|21:08:59.284] Starting MConnection                         module=p2p peer=178.128.206.155:26656 impl=MConn{178.128.206.155:26656}
I[07-16|21:08:59.284] Added peer                                   module=p2p peer="Peer{MConn{178.128.206.155:26656} 5922bf29b48a18c2300b85cc53f424fce23927ab out}"
E[07-16|21:08:59.474] Connection failed @ recvRoutine (reading byte) module=p2p peer=178.128.206.155:26656 conn=MConn{178.128.206.155:26656} err=EOF
I[07-16|21:08:59.474] Stopping MConnection                         module=p2p peer=178.128.206.155:26656 impl=MConn{178.128.206.155:26656}
E[07-16|21:08:59.474] Stopping peer for error                      module=p2p peer="Peer{MConn{178.128.206.155:26656} 5922bf29b48a18c2300b85cc53f424fce23927ab out}" err=EOF
I[07-16|21:08:59.474] Stopping Peer                                module=p2p peer=178.128.206.155:26656 impl="Peer{MConn{178.128.206.155:26656} 5922bf29b48a18c2300b85cc53f424fce23927ab out}"
I[07-16|21:08:59.484] Stopping gossipVotesRoutine for peer         module=consensus peer="Peer{MConn{178.128.206.155:26656} 5922bf29b48a18c2300b85cc53f424fce23927ab out}"
I[07-16|21:08:59.484] Stopping gossipDataRoutine for peer          module=consensus peer="Peer{MConn{178.128.206.155:26656} 5922bf29b48a18c2300b85cc53f424fce23927ab out}"

Ideally we could have some way for a seed node to close the connection gracefully, which can be detected by the connecting node.

[ValarDragon]

I believe the error is occuring at https://github.com/tendermint/tendermint/blob/master/p2p/conn/connection.go#L484

The connection doesn't have access to the p2p config (nor should it) to check if the node its connected to is a seed node. I think we should add a boolean seed value into the connection config. I'll wait for more feedback before changing any code.

[unclezoro]

So what's the plan to fix this issue?

[ebuchman]

So I did some work on this in #2802, but that just ensures the msgs get sent before the conn is close. It still results in the EOF error messages.

How should we handle this? Do we just not log an Error if the conn closed with EOF, but instead an Info that the connection closed? Or are we supposed to introduce some kind of message which says we're going to terminate the connection, so the peer expects it? Or do we just find a way to check if the peer is a seed, in which case we expect this EOF and we log it as Info instead of Error?

[cmwaters]

This behavior is no longer present in the new seed node so I am closing this issue