Decide if we need shorter timeouts on PeerSet fanout requests

[teor2345]

TODO

After #2214 is done, check if Zebra still has this issue, and decide how important this ticket is.

Motivation

Zebra does a lot of PeerSet fanouts that never get a reply.

This can contribute to CI failures like:

test activate_mempool_mainnet ... FAILED

Dec 02 21:33:44.650 INFO zebrad::components::mempool::crawler: Failed to crawl peer for mempool transactions: request timed out
Dec 02 21:33:44.650 INFO zebrad::components::mempool::crawler: Failed to crawl peer for mempool transactions: request timed out
Dec 02 21:33:44.650 INFO zebrad::components::mempool::crawler: Failed to crawl peer for mempool transactions: request timed out

0: stdout of command did not contain any matches for the given regex
"stopping at configured height"

https://github.com/ZcashFoundation/zebra/runs/4400724206?check_suite_focus=true#step:14:1727

https://github.com/ZcashFoundation/zebra/runs/4400724206?check_suite_focus=true#step:14:1727

Designs

• Put a shorter timeout on PeerSet fanouts - see the CandidateSet::update timeout for an example

Related Work

• Regularly yield to other zebrad async tasks #3113

[teor2345]

This bug contributed to this CI failure:
https://github.com/ZcashFoundation/zebra/runs/4408845461?check_suite_focus=true#step:14:1235

[conradoplg]

Also here apparently https://github.com/ZcashFoundation/zebra/runs/4430789103?check_suite_focus=true#step:14:1221

[teor2345]

There are 4 fanouts in Zebra:

• mempool
• sync obtain tips
• sync extend tips
• candidate set addresses (already has a shorter timeout)

You can see them all here - you might need to search in each file:
https://github.com/ZcashFoundation/zebra/search?q=FANOUT

As part of this PR, we need to test that the peers become ready after the shorter timeout. I think there might be some peer set hang bugs in Zebra, but PR #3200 should help fix or diagnose them.

[mpguerra]

Hey team! Please add your planning poker estimate with ZenHub @conradoplg @dconnolly @jvff @oxarbitrage @teor2345 @upbqdn

[teor2345]

We should fix the underlying PeerSet bugs, rather than trying to work around them using timeouts. (Because timeouts can cause other bugs.)