feat: message prioritization with immediate peer-published dispatch and queuing for other msgs

[diegomrsantos]

Motivation: We are assuming the local peer might be spending too much time/bandwidth relaying messages rather than publishing its own messages, which could increase latency for the latter.

Objective: To enhance efficiency in message processing by prioritizing immediate dispatch of peer-published messages, and managing other messages through a queuing system.

Changes:

Immediate Dispatch for Peer-Published Messages: Ensures top priority by immediately sending messages published by the peer.
Queued Handling for IWANT Replies and Relay Messages: Introduces a unified queuing system for both replies to "IWANT" messages and messages from other peers that need relaying. These are processed after peer-published messages.
Asynchronous Task Queue Tracking: Implements an additional queue to monitor asynchronous tasks associated with each published message. The queue for IWANT replies and relay messages is processed once this is cleared.

[codecov]

Codecov Report

Attention: 6 lines in your changes are missing coverage. Please review.

Comparison is base (e0f70b7) 83.17% compared to head (ace2c68) 83.28%.

Additional details and impacted files

@@             Coverage Diff              @@
##           unstable    #1015      +/-   ##
============================================
+ Coverage     83.17%   83.28%   +0.11%     
============================================
  Files            91       91              
  Lines         15351    15442      +91     
============================================
+ Hits          12768    12861      +93     
+ Misses         2583     2581       -2     

Files	Coverage Δ
libp2p/protocols/pubsub/floodsub.nim	90.00% <100.00%> (ø)
libp2p/protocols/pubsub/gossipsub.nim	88.22% <100.00%> (+0.13%)	⬆️
libp2p/protocols/pubsub/pubsub.nim	85.44% <100.00%> (+1.53%)	⬆️
libp2p/protocols/pubsub/gossipsub/behavior.nim	88.39% <75.00%> (ø)
libp2p/protocols/pubsub/pubsubpeer.nim	93.49% <93.82%> (+0.99%)	⬆️

... and 4 files with indirect coverage changes

[kaiserd]

Thank you :). Looks good to me.
Did you already test this in a test network / simulation, to make sure the behaviour is as expected.

Why did you open a new PR? Wouldn't it be easier to track changes if we'd push new commits to the existing ones?

[diegomrsantos]

@kaiserd I tested it running a local Nimbus node on Holenski, the previous metrics seem fine. But I see huge spikes in the non-priority queue size (can be hundreds of thousands for some nodes. I'm subscribing to all subnets to have enough traffic. It'd be good to test with a validator node as well.

[diegomrsantos]

My idea with this PR was to make the difference between the implementations more explicit and potentially testing each one easier. I'm gonna squash the commits here and close the original PR later.

[kaiserd]

Thank for the clarification.

For information of contributors not in the Vac P2P unit: We discussed in the meeting this is experimental and not meant for merging yet. We'll also do more experiments on #1009
We will perform more testing on this, including larger scale tests.

[kaiserd]

LGTM. Thank you :).