fix(share/discovery)!: revamp Discovery

[distractedm1nd]

Closes #2107

There was a discrepancy between the amount of peers inside of the peer manager full node pool, and the limitedSet inside of discovery. It was possible for the node to get in a state where it had no more full nodes to sample from, because the peers inside the limitedSet were blocking on the connection, and never being handed off to the OnPeersUpdate callback.

To fix this, we (@Wondertan, @walldiss and I):

• Set a context deadline for the call to Connect with a peer. Previously, it would never timeout. This is because RoutedHost will block until the context is canceled, even though there is a peer dial timeout.
• Only add peers to the limitedSet after we have successfully connected. This turns the previous peerLimit into a "soft" limit. This prevents in-progress connections from clogging spots in the limitedSet, but also allows for more peers to end up in the limited set than the set limit (we don't throw away already connected peers).
• Removed the timer reset upon a peer disconnecting. This peer disconnection happens often, causing a significant delay in the next call to FindPeers.
• Added logs, cleaned up various comments
• Backoff
  • Adds a RemoveBackoff method to the backoffConnector, to remove cancelled connections from the cache
  • Adds Backoff method
• Cleans up limited set
• Added tests for Discovery
• Handle the case where discovered peer is already connected
• Introduces importable Discovery params
• Revisits constans

I have tested this locally, and it works well with various peer limits. The first few calls to findPeers cause the highest peer count in the limited set (roughly about double), but bad connections are trimmed relatively quickly, and within a few cycles, we land with a pretty stable peer set where there are only 1-2 peer changes every interval, if any.

NOTE: Config breaking!

[renaynay]

findCancel and cancelFind - lets pick one

[Wondertan]

I will also write tests over the weekends and maybe refactor if enough inspiration

[codecov-commenter]

Codecov Report

Merging #2117 (f837321) into main (c4e994b) will decrease coverage by 0.43%.
The diff coverage is 47.00%.

@@            Coverage Diff             @@
##             main    #2117      +/-   ##
==========================================
- Coverage   55.80%   55.37%   -0.43%     
==========================================
  Files         209      212       +3     
  Lines       13289    13643     +354     
==========================================
+ Hits         7416     7555     +139     
- Misses       5131     5327     +196     
- Partials      742      761      +19     

Impacted Files	Coverage Δ
api/gateway/availability.go	0.00% <0.00%> (ø)
api/gateway/header.go	0.00% <0.00%> (ø)
api/gateway/share.go	8.91% <0.00%> (-0.77%)	⬇️
das/metrics.go	45.53% <0.00%> (ø)
share/p2p/peers/metrics.go	32.48% <32.48%> (ø)
share/p2p/shrexnd/client.go	68.33% <33.33%> (+5.83%)	⬆️
share/availability/discovery/discovery.go	53.17% <43.93%> (-3.98%)	⬇️
share/eds/store.go	67.80% <50.00%> (-0.78%)	⬇️
share/availability/discovery/backoff.go	79.24% <55.55%> (-5.20%)	⬇️
share/getters/shrex.go	85.57% <57.14%> (-2.18%)	⬇️
... and 11 more

... and 5 files with indirect coverage changes

[renaynay]

only one comment re panic

[walldiss]

Wow 58 commits! Discovery is so much cleaner now!

[distractedm1nd]

Blocked because @vgonkivs has previously requested changes