improved peer ranking

Robust peer ranking by delivery quality

The goal is to rank upstream peers by how quickly they will actually deliver what we fetch — EB bodies, closures — so fetch logic can route to the best and demote the worst (feeding the improvements proposed in leios-fetch-scheme.md — Improvement #1's peer classification, adaptive chunk sizing, and the like). The ground truth is directly-measured completion time — T_wait, the observed request-to-full-reply timing — and where a fetch has completed we rank on it: a paced-slow peer measures slow — its delivery is exactly what T_wait records — so the signal can't be gamed. Its one limitation is that it is retrospective: you cannot measure a fetch you have not made, so to choose a peer before the fetch, and for chunk-sizing and diagnostics, we need a forward prediction — a time-to-deliver estimate, ramp(RTT) + estBurstS × size, built from network-mux's DeltaQ/RTT instrumentation (estBurstS, seconds/byte, from its burst-SDU serialisation estimator; RTT from its cookie-echo window; a branch feature today, not yet on main). Reusing instrumentation the mux already provides, rather than new plumbing, has a catch: the throughput input estBurstS is adversarially gameable — a peer can pace its SDUs so estBurstS reads fast (high apparent bandwidth) while it dribbles our data, earning a rank it doesn't deserve and winning traffic it then serves slowly. (RTT is manipulable too, but asymmetrically: a peer can't fake the cookie-echo RTT low — it can't echo faster than the round-trip — so cheap manipulation only inflates it, which self-penalises by raising its time-to-deliver. Lowering its actual RTT means co-locating near the victim: real, but expensive, where gaming estBurstS is free. A co-located peer that looks genuinely fast while dribbling the payload is caught by the actual-vs-predicted per-fetch check, not by RTT.)

So the prediction has to be robust: a peer must not be able to look good in the estimate while delivering badly. This document lays out how — a promptness-gated burst window that yields a trustworthy estBurstS, why ranking on time-to-deliver makes a gamer self-penalise, and an actual-vs-predicted backstop that audits the estimate against measured T_wait and churns consistently-off peers. The hierarchy throughout is that direct measured completion is the preferred basis: the peer ranking — the very decision a pacer sets out to game — rests on measured completion wherever a fetch has completed, and there pacing gains nothing. The estBurstS hardening exists to make the prediction trustworthy for the roles measurement can't fill: choosing a peer before a fetch, chunk-sizing, slow-loris, and diagnostics.

---

The attacks we consider

All four deny us a timely fetch — an EB body or closure that doesn't arrive in time for the round's diffusion/certification deadline — cheaply for the adversary. Three of them win by looking good enough to be chosen and then not delivering; slow-loris simply drip-feeds a fetch it already holds. They differ in how they're caught.

• RTT manipulation. Make apparent RTT low to score well on the ramp term. A peer can't fake the cookie-echo RTT low — it can't echo faster than the round-trip — but it can buy a low real RTT by co-locating near us (expensive), or inflate its RTT by delaying echoes, which only self-penalises. Caught by: the ranking (inflation self-penalises); a co-located peer that then dribbles falls to the per-fetch check. → Using the RTT signals.
• Pacing (idle-gap). Deliver in tight micro-bursts split by silent stalls, so a gap-based estBurstS counts only the fast bursts and reads high while delivery is slow. Gain: fake high bandwidth → win the fetch → dribble the payload. Caught by: the promptness-gated burst window (silence forfeits the extension), reinforced by the estBurstS/RTT bind. → The window-sizing tension; The adversary bind.
• Byte-padding. Serve other bytes fast (other requests / filler) so the aggregate estBurstS and RTT look great, while dribbling our closure — targeted withholding (the data-withholding threats T20/T21). Caught by: per-fetch completion timing and the actual-vs-predicted check, churned on persistence. → The adversary bind (byte-padding).
• Slow-loris. Accept the fetch and drip-feed it — no attempt to hide; it ties up our fetch slot and, at scale, our concurrent-fetch capacity, denying us throughput cheaply (we wait on it instead of a good peer). Unlike pacing it doesn't fake fast — it looks slow. Caught by: the ranking (it scores low), throughput-collapse detection (estBurstS rising), and cancelling/reissuing the stalled fetch. → The bigger picture.

The rest of this document is the machinery behind those mitigations.

The window-sizing tension, and the gated extension that resolves it

Requests are small — often a single SDU — but responses arrive as a burst of many SDUs, and the cookie on our request is echoed back on that burst. The first response SDU's echo gives the RTT sample; the follow-up SDUs of the same burst feed estBurstS, the serialisation rate. That poses the question the burst window exists to answer: for how long do we keep attributing incoming SDUs to this response's burst — counting them toward its throughput — before a new SDU counts as a fresh exchange instead? That horizon is rttBurstMaxAge (hard-coded 1 s today).

It has to be two things at once: tight enough that a pacer can't stall-then-look-fast, and extensible enough to measure a genuine multi-second closure. A fixed 1 s is a poor compromise — too short for a large honest transfer, too generous to a pacer if loosened.

When a burst would otherwise time out mid-transfer there are two choices: stop counting (and leave honest data on the table if the peer was good and the stream healthy), or keep extending the deadline while the peer is still actively serving. The second is the gate — extend while the peer serves promptly, end the window when it goes quiet.

"Serving promptly" can be read off network-mux's cookie mechanism (a branch feature today, not yet on main): we piggyback cookies on our outbound SDUs, the peer echoes them back, and the still-unechoed cookies sit in a queue — the cookie PSQ — whose matched echoes yield the RTT samples. Repurposing that queue as a liveness gauge is the new part: a peer echoing promptly keeps its oldest outstanding cookie young; a peer gone quiet lets the oldest cookie age and the queue grow. So there are two readings of "serving promptly":

• Cookie echoes (general): extend while the cookie queue isn't growing / its oldest cookie isn't aging.
• Incoming SDU spacing (one-directional download, where our cookie-carrying outbound is sparse): extend while successive SDUs arrive within a tight back-to-back max-spacing.

Either way an honest large closure keeps the window open (its SDUs / echoes keep coming) while a pacer that goes silent to game the estimate forfeits the extension. The tight spacing has one honest hazard — a loss / RTO briefly stalls the in-order stream and can trip it — so the anti-gaming does not rest on the gate alone: it rests on the actual-vs-predicted backstop (below), which tolerates a one-off trip and churns only a peer whose delivery is consistently worse than its advertised rate.

Using the RTT signals — and a circularity to avoid

The queue is a liveness gate on the burst extension (above): a trend — is the peer serving now — read from whether the sent-cookie queue is growing / its oldest cookie is aging. It is deliberately not a comparison of queue duration against the self-measured RTT quantile: that would be circular — an adversary delaying echoes uniformly inflates the queue front-age and the measured RTT together (a cookie echoed at t0 + RTT + D gives both a front-age peak and an RTT sample of RTT + D), so the comparison never trips. The trend has no such weakness: a silent peer's queue grows whatever its RTT was.

The self-measured RTT is used heavily — as an input, not against itself. The idle-gap gamer's inflated RTT feeds the time-to-deliver ranking (ramp(RTT) + estBurstS × size) and self-penalises: high apparent RTT ⇒ high time-to-deliver ⇒ ranked down, no independent reference needed (the bind and bigger-picture sections, below). Busy-fill — keeping the burst alive with filler SDUs rather than going silent — is caught the other way: the filler is counted, so estBurstS reads the true, slow rate and the estimate ranks the peer down. (Filling fast with non-closure bytes to hold estBurstS low instead is a different attack, byte-padding — the one escape these self-measured signals miss, caught by the per-fetch check; see the bind.) So the scheme deprioritises a gamer on self-measured signals alone.

An independent RTT reference buys cleaner attribution, not the core defence. A kernel tcpi_rtt or handshake baseline gives a single-observation tell of adversarial app-layer echo-holding (kernel RTT low, app-RTT high ⇒ the peer is sitting on cookies — literally the existing "app-RTT vs kernel-RTT divergence" cross-check), distinguishing it from an honestly far peer (both high). That matters only for escalating from deprioritise — which self-measured signals already do — to demote/blocklist.

Attribution isn't gated on it, though: the actual-vs-predicted comparison (below) recovers a form of it portably — a persistent actual ≫ predicted identifies targeted withholding using only app-layer signals (direct T_wait, estBurstS, RTT), no kernel/handshake reference. It convicts more slowly (it needs persistence to clear the serve_time/cold-cache confound) and it catches the attack by a different tell.

The two attacks leak differently — idle-gap through RTT, byte-padding through the payload — as the bind section below sets out. So the independent reference's portability caveats (Linux + tracetcpinfo, design-phase, for tcpi_rtt; drift-prone for the handshake baseline) cost only the fast, clean attribution signal — not attribution outright, and certainly not the core defence.

The adversary bind — and why aggregate estBurstS is a strength

What makes the gate sound is a genuine dilemma the adversary faces on a single measured stream:

• To make delivery slow, the stream must be idle a lot (low byte-rate).
• To game a gap-based estBurstS, that idle must fall outside bursts (the burst timer expires during silence, so the stall isn't counted).
• But idle = no SDUs = no echoes → the cookie queue ages / RTT samples inflate.
• To keep echoes prompt, it must fill the idle with SDUs — but then the burst stays alive, the fill is folded into estBurstS, and the estimate reads the true (slow) rate.

So it cannot simultaneously game a gap-based throughput estimate and keep the liveness/echo signal flowing: whichever activity carries the prompt echoes is measured, and going silent to hide the stall shows up as echo lag. Pick one, caught by the other.

Cardano's mini-protocols are mostly pull-based, which tightens this further: the peer sends SDUs only in response to our requests, so it cannot inject filler at will to keep echoes flowing. The fill would have to be responses to some other request we happen to have outstanding to it — an opportunity it cannot manufacture — so busy-fill is opportunistic and hard to exploit reliably. Absent such traffic, the only way to withhold our closure is to fall silent, which the echo-lag / SDU-spacing gate catches directly.

Aggregate (protocol-mixed) estBurstS is what closes the loophole. The bind holds only if the echo-carrying SDUs are inside the measurement — and today estBurstS is connection-level, so every received SDU, whatever mini-protocol carries the echo, feeds the aggregate burst. There is no unmeasured side-channel to park prompt echoes on. The protocol-mixed aggregate we usually treat as a limitation is, for this attack, the strength.

Caveat for a future per-protocol split. The loophole reopens if per-protocol estBurstS (the proposed "Shape B" split — per-mini-protocol DeltaQ buckets, not implemented today) is added without also binding the cookie/RTT per-protocol — because then the throughput signal and the echo signal live at different granularities. Concretely: we fetch a 12 MB closure over LeiosFetch (protocol B) and are also driving a second mini-protocol A (keep-alives, say, or another fetch) on the same connection.

• It delivers the closure in tight 50 kB micro-bursts, then goes silent on B for ~500 ms, repeat. Per-protocol estBurstS_B counts only the fast micro-bursts — the 500 ms silences exceed B's burst gap and fall outside its bursts — so it reads fast. Gamed.
• During each B-silence the peer keeps answering our A requests promptly, each response echoing our latest cookie. The cookie PSQ is connection-level, so those A echoes keep the queue drained and RTT low — the gate stays quiet.

Result: estBurstS_B says LeiosFetch is fast and the gate says nothing is held back → we rank the peer a great closure-deliverer, route closures to it, and it dribbles them. A is a side-channel that estBurstS_B doesn't see, yet it still satisfies the connection-level gate — exactly the gap the bind depends on being closed. The loophole is gated on A traffic frequent enough to keep draining the connection-level queue through the B-silences, though — consistent with the pull-based limit above, the peer can't fabricate that; it can only ride A traffic we ourselves drive.

The aggregate estimate doesn't have this gap: those same A responses are inside the aggregate burst, so they either keep it alive (⇒ estBurstS reads the slow closure rate) or, if the peer goes fully silent to avoid that, the echoes lag (⇒ caught). The fix under a per-protocol split is to keep the two signals at the same granularity — judge B's delivery against echoes that ride B, so a silence on B costs echo-lag on B. Rule: go per-protocol on estBurstS ⇒ go per-protocol on the echo/RTT too.

Where the payoff hides — and how each strategy leaks. The adversary's payoff is always the same: pace the bursts so estBurstS reads fast, win the fetch, then dribble the payload. Stalling echoes or inflating RTT is never the goal — it is only a leak of the pacing, and a different signal catches each leak.

• Busy-fill — fill the stalls with SDUs so the echoes stay prompt and nothing looks held back. But those filler SDUs keep the burst alive, so they fold into estBurstS, which then reads the honest, slow rate. estBurstS itself catches it.
• Idle-gap — go silent between micro-bursts so the stalls fall outside the burst and estBurstS reads fast. That same silence starves the echoes: the cookie front-age grows and the RTT-quantile inflates. RTT catches it — self-penalising in the ranking, and, against an independent reference (kernel tcpi_rtt / handshake), attributable as app-layer echo-holding.

(The queue-vs-self-RTT comparison stays circular — uniform echo delay inflates the front-age and the RTT sample together — but the bind means we never need it.) The third leak, byte-padding, escapes both of these and is treated next.

The one escape the bind does not close: byte-padding. The peer sends other bytes fast, serving another protocol request, so the aggregate byte-rate looks high while it dribbles the closure you actually asked for. estBurstS reads fast, the padding carries prompt echoes (the gate stays quiet), yet your closure is slow. A measurement bound to that specific fetch — direct completion timing (T_wait on the closure; next section) — catches it, because the padding isn't the bytes you're waiting on. And comparing that T_wait to what the peer's own signals predict — ramp(RTT) + estBurstS × size — upgrades detection to attribution: under padding the prediction is fast but the actual is slow, and actual ≫ predicted means the peer has the capacity (it's spending it elsewhere) and is withholding from us (the data-withholding threats T20/T21), not merely slow. Lose-lose for the padder: advertise fast and dribble ours ⇒ discrepancy; advertise slow to match ⇒ ranked down anyway; deliver ours fast ⇒ not the attack. Caveat: honest confounds fire the same discrepancy — server-side serve time for our closure, HOL/contention on the shared bearer, stale estBurstS/RTT — so deprioritise on a one-off and punish (demote/blocklist) only on persistence with those ruled out.

The serve-time confound is the sharpest, because it produces the same observable, not just a noisier one. estBurstS/RTT capture the byte-flow rate and the network round-trip, but not the peer's cost to produce the bytes — reading our closure's txs from its store, reassembling it:

actual    = RTT + serve_time + size × per-fetch-rate + loss/RTO
predicted = RTT +              size × estBurstS       (aggregate signals)

The prediction has no serve_time term, so actual − predicted ≈ serve_time: a large serve time alone reads as actual ≫ predicted. So a peer whose TxCache holds others' closures (hot, served fast) but not ours (cold, disk-read) shows a fast aggregate and a slow closure — capacity that exists but isn't reaching our request, exactly like byte-padding, yet benign (which closures are hot/cold is diffusion history, not the peer choosing to starve us). Decomposing our fetch into time-to-first-byte (which absorbs serve time) vs the whole-fetch byte-rate separates the two if the peer batch-reads (long TTFB, then a normal rate ≠ dribbling) — but a slow streaming read shows a slow rate throughout and is indistinguishable from withholding by rate alone. So the discrepancy stays a deprioritise signal (correct either way — the peer is slow for us now); promoting it to malice needs the slowness to be systematic across our requests to that peer (targeting), not per-request random (cache-luck).

The queue is the structure — the extension gate

The cookie/promptness queue is the load-bearing structure: the gate that lets the burst window extend to measure an honest large closure while denying a pacer the same extension (above). The consistency and actual-vs-predicted checks sit on top of it as the backstop, not as a substitute for it. It reuses machinery already present — the t-digest and the cookie PSQ; what is new is gating the extension on them.

Setting the threshold for the actual-vs-predicted check. For the byte-padding check (how much may actual exceed predicted before we act), don't derive the band from estDeltaQVVar. It's the wrong granularity — per-SDU micro-jitter, which ignores the correlated per-fetch terms (ramp-round count, loss-recovery, serve_time), so it under-estimates the real spread and the band comes out far too tight — and the wrong shape, since the residual is heavy-tailed and a σ-band mis-fits it. Threshold instead against an empirical high quantile (say p99) of the residual actual − predicted, from a population baseline across honest peers — not the suspect's own samples, which it can inflate to widen its band and hide under.

Stratify that baseline by regime, because the residual's spread is not one distribution. It scales with message size (more bytes ⇒ more loss-recovery exposure and a longer transfer) and with RTT (both the ramp-round count and the RTO magnitude grow with it — and that dependence survives even though predicted already subtracts a ramp(RTT) term). Loss rate is the natural third axis, but it is non-portable — only via kernel tcpi_retrans/tcpi_lost (Linux + tracetcpinfo, design-phase) — so a first pass stratifies on size × RTT only and pools clean and lossy paths in the same bin. The cost is a deliberately conservative band: the lossy fetches in a bin inflate its p99, widening the threshold, so first-pass detection is less sensitive (a padder can hide under the loss-inflated band) but rarely false-positives. Add the loss stratum to tighten it once kernel telemetry is available on the measuring node.

The bigger picture: rank on measured completion, not estBurstS

The pacing attack only matters to the extent we use estBurstS as a proxy for delivery quality (peer ranking, chunk sizing). But those decisions are moving onto direct completion timing (T_wait / observed arrival). Against direct completion timing, pacing is self-defeating: the adversary's paced-slow delivery is the measured completion time, so it ranks low no matter how it games the per-SDU gaps. So the highest-leverage defense against estBurstS-pacing is not to trust estBurstS for the decision that pacing targets — rank on observed completion, and pacing buys the adversary nothing.

Even short of direct timing, ranking on estimated time-to-deliver — ramp(PeerRTT) + size × estBurstS (Improvement #1's fused metric) — already absorbs the idle-gap gamer, which self-penalizes: faking a low estBurstS by going silent forces its apparent RTT up (held cookies), and the RTT term then dominates its estimate, so a genuine low-RTT/high-throughput peer always out-ranks it (higher apparent RTT ⇒ higher time-to-deliver). This needs no independent cross-check — you never have to distinguish the gamer from a genuinely far peer, you just rank both down for their RTT; the tcpi_rtt/handshake anchor is only for attributing and punishing the gamer specifically. Two caveats: the ranking RTT-quantile must be high enough to capture the stall inflation (occasional long stalls hide above p80), and byte-padding evades it — it shows low RTT (see the bind). So the ordering is: measured completion timing closes both; the fused estimate is a good fallback that handles idle-gap for free.

That leaves estBurstS needing hardening only for its residual uses — diagnostic decomposition, and slow-loris detection. Slow-loris is the mirror image of pacing: the peer makes no attempt to hide its slowness, it just accepts the fetch and trickles it, tying up our slot. Because it looks slow, the time-to-deliver ranking deprioritises it for free; and a peer whose estBurstS rises mid-transfer — throughput collapsing below a floor — is the slow-loris signature (Improvement #3), at which point we cancel the stalled fetch and reissue it elsewhere. So estBurstS earns its keep here in the opposite direction from pacing — flagging a peer that looks slow, not one faking fast — where pacing-to-look-fast can't help the adversary. The estBurstS-vs-RTT consistency check, meanwhile, is worth keeping as a sanity gate for when you do lean on estBurstS, but it shouldn't be load-bearing.

Why leaning on estBurstS is sound

estBurstS is low-level and gameable, so building the prediction on it looks risky — but it's safe because of the role it plays:

• Center-line, not tolerance. It supplies the peer's advertised rate in ramp(RTT) + estBurstS × size. The tolerance — how far actual may exceed that — is a separate quantity: a high quantile (p99) of the residual actual − predicted measured across honest peers (the threshold discussion above). Call that honest-peer residual distribution the baseline. estBurstS's missing macro terms (serve_time, stalls, loss-recovery) don't corrupt the check: because predicted omits them, they surface in honest peers' residuals too, so they're already folded into the baseline — a suspect peer is flagged only by deviating above it.
• Audited, not trusted. actual (direct T_wait) is the auditor; predicted (from estBurstS) is the claim under audit — its unreliability is fine because it's the thing being checked, not the ground truth.
• Inaccuracy cancels; gaming self-defeats. Because that baseline is computed with the same estBurstS, any systematic bias in estBurstS (say it reads 10% fast for everyone) shifts every peer's predicted alike, shifts the baseline with it, and cancels in the residual-vs-baseline comparison — so the check needs estBurstS consistent across peers, not accurate. And the only useful lie is to make estBurstS look low, which shrinks predicted and makes the check more sensitive; lying it high advertises slow and ranks the peer out.

The one role estBurstS is not given — trusted ground truth for ranking — is the one it couldn't survive; that's why ranking ultimately prefers measured time-to-deliver. Correspondingly, the tolerance band is the empirical residual, not the per-SDU estDeltaQVVar (the threshold discussion above).

Smaller points

• Your own GC is a confound for the gate. A GC pause makes you timestamp echoes (and incoming SDUs) late, inflating the cookie front-age and the SDU-spacing reading — looking like the peer stalled, a false extension-revocation that blames the peer for your own pause. Same class of false trip as a loss/RTO on the gate (the window-sizing section); the backstop's persistence requirement absorbs it. Kernel tcpi_rtt is immune to your GC — a reason to prefer it for the RTT input where available.
• Lazy sampling The RTT quantile feeding ranking/prediction is read from the t-digest every ~1 s in processIngress, not per-SDU — preserving the finger-tree's deferred <>. (The extension gate itself is a cheap front-age / SDU-spacing check, separate from the t-digest.)
• Not tuning-free. The knobs are the promptness threshold (the tight back-to-back SDU spacing / front-age), the backstop quantile (p99), and the churn-persistence count. The tight threshold's risk is false trips (loss/RTO, GC), not gaming — a steady-dribbler that stays just under it keeps the burst extending and is then measured slow, which is correct — and those false trips are what the backstop's persistence tolerates. The knobs are physically grounded (a spacing in milliseconds, an RTT, a percentile), but they still need calibration.
• Download liveness needs no cookies. In a one-directional download the incoming-SDU-spacing gate supplies liveness directly; the cookie signal (for RTT) can go stale there — cookie-carrying outbound is sparse — so inject a cookie periodically if you need a fresh RTT through a long download.
• How the signals relate. We compute estBurstS ourselves from the peer's SDU gaps, so a peer influences it only by pacing its own SDUs; the cookie echoes are an orthogonal measurement we control. Pacing the SDUs and starving the echoes coincide only when echoes ride the measured stream — which aggregate estBurstS guarantees (the bind section); a per-protocol split would have to preserve it.
• Response on detection is cheap and low-blast-radius: the ranking already deprioritises a gamer; on a persistent actual-vs-predicted discrepancy, churn/demote it. It's a peer trying to look good, so a false demotion costs little.

Bottom line

The window-sizing tension — tight enough to deny a pacer, extensible enough to measure a large closure — is resolved by gating the burst extension on peer promptness (cookie-echo front-age, or incoming SDU spacing on a download): an honest large transfer keeps the window open, a pacer's silence closes it. The gate is imperfect alone — a loss/RTO can trip the tight spacing, and steady-dribble or byte-padding slip through — so the anti-gaming rests on the actual-vs-predicted backstop: churn a peer whose delivery is consistently worse than its advertised rate, which also absorbs the gate's false trips via the consistency requirement. An independent RTT reference (kernel tcpi_rtt / handshake) is a nice-to-have for fast attribution, not a prerequisite. And all of this hardens the prediction: the ranking itself prefers directly-measured completion wherever a fetch has finished, where a pacer gains nothing.