Fix Issues With Peer Handshakes

[nisdas]

What type of PR is this?

Bug Fix

What does this PR do? Why is it needed?

• Does not perform status requests on inbound dials, instead waiting for the peer
  to send their status request and then appropriately responding.
• Prevents multiple handshakes happening for the same peer at once to prevent multiple redundant goodbyes.
• We add a small delay for each goodbye, so that it gives enough time for the remote peer to be able to read the message.

Which issues(s) does this PR fix?

Fixes #5741 #5796

Other notes for review

[farazdagi]

LGTM.
Couple of minor things (feel free to ignore):

• magic constants (like 10sec to wait for peer to handshake, 50 seconds to flush the stream)
• not sure if repeatPeerConnections is used anywhere. It is declared in monigoring.go and then incremented, but not sure whether its data is used anywhere.

[terencechain]

I realize the usual lack of tests for p2p related handles, is this another instance that's due to libp2p?

[nisdas]

@farazdagi alright changed it so as to declare them as constants first. Although I am not sure on this point.

not sure if repeatPeerConnections is used anywhere. It is declared in monigoring.go and then incremented, but not sure whether its data is used anywhere.

This is for our prometheus metrics collection, it usually isn't used anywhere else.

@terencechain Yeap this is a bit complicated to do, as there as many moving parts to this. The handshake process involves code from sync too and has a lot of pre-conditions which will be difficult to write a test for. On top of that testing libp2p related paths are non-trivial.

[codecov]

Codecov Report

Merging #5799 into master will increase coverage by 0.07%.
The diff coverage is 11.42%.

@@            Coverage Diff             @@
##           master    #5799      +/-   ##
==========================================
+ Coverage   59.75%   59.83%   +0.07%     
==========================================
  Files         311      311              
  Lines       26148    26108      -40     
==========================================
- Hits        15626    15621       -5     
+ Misses       8392     8358      -34     
+ Partials     2130     2129       -1