Repository navigation
P99 spikes on a fan-in mpsc consumer — is the LIFO slot starving my producers, or is try_recv bypassing coop?
#8498
|
Runtime: ~2,000 short-lived tasks Median latency (timestamp taken just before Consumer: tokio::spawn(async move {
loop {
// drain as much as possible before doing downstream work
let mut batch = Vec::with_capacity(256);
while let Ok(item) = rx.try_recv() {
batch.push(item);
if batch.len() == 256 { break; }
}
if batch.is_empty() {
match rx.recv().await {
Some(item) => batch.push(item),
None => break,
}
}
process_batch(&mut batch).await;
}
});What I found reading the source:
Questions:
|
Replies: 2 comments 2 replies
|
Yes, Your code looks like a manual implementation of |
|
Part of your hypothesis is right and part isn't, and the part that isn't matters for choosing the fix. Short version: 1.
|
| Suspect | Verdict |
|---|---|
try_recv bypasses coop |
True, but ~µs of work at a 256 cap. Not your 10–50ms. |
| Consumer monopolizes LIFO slot | No: LIFO shares the prior task's budget and is capped at 3 consecutive polls. |
| Consumer stranded in a non-stealable LIFO slot behind a slow poll | Plausible. Confirm via scheduled time; mitigate with short polls or disable_lifo_slot() (unstable). |
Bounded-channel backpressure during slow process_batch |
Plausible, and often the real answer for fan-in. Measure send().await wait. |
Switch to recv_many, measure with tokio-metrics, and let the numbers tell you whether you're looking at scheduling delay or queueing delay before touching runtime configuration.
Part of your hypothesis is right and part isn't, and the part that isn't matters for choosing the fix. Short version:
try_recvdoes bypass coop, but a 256-itemtry_recvloop is microseconds of work and can't by itself produce 10–50ms stalls. The LIFO slot is a plausible culprit, but not through the mechanism you describe. And there's a third candidate you haven't ruled out: backpressure on the bounded channel itself.1.
try_recv()and the coop budgetYes, it bypasses coop. The async path goes through
Chan::recv(cx), which starts with:try_recv()…