Peer connection test cases

[kilrau]

This issue is to collect a set of peer connection test cases with the goal to reliably reproduce issues regarding connecting/re-connecting to peers that continue to pop up: #2009 (comment)

[sangaman]

In my tests, I encountered the following behavior that I believe is directly related to the comment linked above.

1. Start xud and connect to an outbound peer.
2. Turn off wifi and wait a bit, peer remains in the local xud list. This is an issue in itself that requires some more investigation, but I'm guessing this has to do with the way TCP is designed.
3. Restart xud and re-enable wifi. Reconnection attempts happen but they get rejected because the peer thinks it is already connected to us and does not accept the connection. This happens in a loop.

I'll look into why these "zombie" peers are remaining in the peers list even when connectivity is lost. Our ping/pong packets should be disconnecting these peers when the ping doesn't receive a corresponding pong, so my first thought is that something isn't working correctly there.

As a quick overview of the behavior we expect:

So can anyone please describe how P2P connection establishing / connection supporting / connection restoring cases work in xud?

Nodes advertise listening addresses and gossip other known nodes and their respective listening addresses. A given node will listen for inbound peers on its inbound addresses and also attempt outbound connections for known nodes with listening addresses. Once connected, nodes exchange ping and pong packets on a timer, and most packets have a time threshold by which they expect a response. If a node expects a response from a peer but doesn't get one in time, it should disconnect that peer and send it a final packet explaining that it's disconnecting due to stalling. In case two peers lose connection (due to stalling, for example) the peer that originally initiated the connection should attempt to reconnect.