Skip to content

improved peer ranking

Jump to bottom
Marcin Wójtowicz edited this page Jul 17, 2026 · 32 revisions

Robust peer ranking by delivery quality

The goal is to rank upstream peers by how quickly they will actually deliver what we fetch — EB bodies, closures — so fetch logic can route to the best and demote the worst (feeding the improvements proposed in leios-fetch-scheme.md — Improvement #1's peer classification, adaptive chunk sizing, and the like). The ground truth is directly-measured completion time — T_wait, the observed request-to-full-reply timing — and where a fetch has completed we rank on it: a paced-slow peer measures slow — its delivery is exactly what T_wait records — so the signal can't be gamed. Its one limitation is that it is retrospective: you cannot measure a fetch you have not made, so to choose a peer before the fetch, and for chunk-sizing and diagnostics, we need a forward prediction — a time-to-deliver estimate, ramp(RTT) + estBurstS × size (hereafter the estimate), built from network-mux's DeltaQ/RTT instrumentation (estBurstS, seconds/byte, from its burst-SDU serialisation estimator; RTT from its cookie-echo window; a branch feature today, not yet on main). Reusing instrumentation the mux already provides, rather than new plumbing, has a catch: the throughput input estBurstS is adversarially gameable — a peer can pace its SDUs so estBurstS reads fast (high apparent bandwidth) while it dribbles our data, earning a rank it doesn't deserve and winning traffic it then serves slowly. (RTT is manipulable too, but asymmetrically: a peer can't fake the cookie-echo RTT low — it can't echo faster than the round-trip — so cheap manipulation only inflates it, which self-penalises by raising its time-to-deliver. Lowering its actual RTT means co-locating near the victim: real, but expensive, where gaming estBurstS is free. A co-located peer that looks genuinely fast while dribbling the payload is caught by the actual-vs-predicted per-fetch check, not by RTT.)

One prerequisite deserves flagging up front: the estimate needs the datum's size before the fetch starts, so the protocol must advertise the size of what it offers (an EB, a closure). In Cardano that is a real gap — virtually no mini-protocol advertises sizes; the lone exception is tx-submission, where the advertised size only drives a penalty when the delivered bytes deviate too far from it. The reason is trust: nodes don't take each other's word, so only independent verification of the downloaded datum counts, and an advertised size is an unverified hint. That hint is a third gameable input to the estimate, and the most direct one, since the peer states it outright — a peer can win the fetch not by being fast but by understating the size (or merely undercutting what competitors advertise), steering us its way. The lie surfaces only after the fact — the delivered size overshoots the advertised one, so actual ≫ predicted — which is exactly how tx-submission already polices it.

So the prediction has to be robust: a peer must not be able to look good in the estimate while delivering badly. This document lays out how — a promptness-gated burst window that yields a trustworthy estBurstS, why ranking on time-to-deliver makes a gamer self-penalise, and an actual-vs-predicted backstop that audits the estimate against measured T_wait and churns consistently-off peers. The hierarchy throughout is that direct measured completion is the preferred basis: the peer ranking — the very decision a pacer sets out to game — rests on measured completion wherever a fetch has completed, and there pacing gains nothing. The estBurstS hardening exists to make the prediction trustworthy for the roles measurement can't fill: choosing a peer before a fetch, chunk-sizing, slow-loris, and diagnostics.

The attacks we consider

All four deny us a timely fetch — an EB body or closure that doesn't arrive in time for the round's diffusion/certification deadline — cheaply for the adversary. Three of them win by looking good enough to be chosen and then not delivering; slow-loris simply drip-feeds a fetch it already holds. They differ in how they're caught.

  • RTT manipulation. Make apparent RTT low to score well on the ramp term. A peer can't fake the cookie-echo RTT low — it can't echo faster than the round-trip — but it can buy a low real RTT by co-locating near us (expensive), or inflate its RTT by delaying echoes, which only self-penalises. Caught by: the ranking (inflation self-penalises); a co-located peer that then dribbles falls to the per-fetch check. → Using the RTT signals.
  • Pacing (idle-gap). Time the delivery so the estimate reads fast — a flatteringly low time-to-deliver — while the closure actually arrives slowly; win the fetch on that estimate, then dribble the payload. Caught by: the promptness-gated burst window and the adversary bind (the timing trick, and why it backfires, are spelled out there). → The window-sizing tension; The adversary bind.
  • Byte-padding. Make the estimate look good by serving other traffic fast, while dribbling the closure we actually asked for — targeted withholding (the data-withholding threats T20/T21). The estimate stays flattering because it reflects the peer's aggregate traffic, not our specific fetch. Caught by: per-fetch completion timing (T_wait) and the actual-vs-predicted check, on persistence. → The adversary bind (byte-padding).
  • Slow-loris. Accept the fetch and drip-feed it — no attempt to hide; it ties up our fetch slot and, at scale, our concurrent-fetch capacity, denying us throughput cheaply (we wait on it instead of a good peer). Unlike pacing it doesn't fake fast — it looks slow. Caught by: the ranking (it scores low) and throughput-collapse detection (estBurstS rising) — though recourse is limited, since no per-fetch cancel exists yet (only wait / reset / hedge, each a trade-off). → The bigger picture.

Those four game a single delivery measurement, and the machinery in the rest of this document is what catches them. A distributed blockchain setting adds three that game the ranking system rather than any one measurement — across time, items, and identities — and these lean on redundancy and peer-selection layers beyond the delivery score:

  • Reputation farming, then a deadline-timed defection. Behave well to earn a top rank cheaply over low-stakes rounds, then withhold the one fetch that matters — an EB near its certification deadline — and revert. Score-based churn reacts only to a sustained low score, so a single withheld body — one non-first-delivery against an hour of farmed good score — barely moves the peer's standing: the betrayal is close to free. The defence is therefore fetch redundancy for consensus-critical items — hedge across peers (Improvement #5) so no single defector is decisive — backed by weighting a consensus-critical miss far above an ordinary one in the score. Redundancy, not punishment, is what protects the fetch.
  • Surgical withholding of the critical item. Byte-padding targets us across a whole connection; the per-item version serves everything fast except the single block/EB an adversary wants suppressed (censorship, fork support). A per-peer average rank barely moves, so the ranking alone won't flag it. The defence is protocol-level: multi-source diffusion fetches the item from someone else, and a missed consensus-critical fetch should weigh far above its share of the peer's history. The ranking's only job is to not rate the withholder top on the strength of the 99% it does serve.
  • Baseline poisoning via Sybils. The actual-vs-predicted threshold is a p99 over a population baseline of honest peers; in a permissionless network an adversary runs many peers and inflates that baseline until a byte-padder fits under the loosened band. The impact is bounded — it desensitises detection, it doesn't falsely accuse honest peers — and it's mitigated by robust, per-subnet-capped aggregation — weight the baseline by IP-prefix / ASN rather than by peer, so a Sybil cluster in one subnet can't dominate it (identities are cheap, distinct network locations are not) — and by capping the band with the estimate itself — a physical model of delivery time — rather than the empirical p99 alone: the plausible residual (serve time, loss-recovery, stalls) has a physical ceiling a poisoned baseline can't inflate past.

A note on the response rather than a distinct attack. In Cardano, churn is score-based: peers first to offer a header or first to serve a requested body score well, and the worst-scoring ~20% are evicted each hour and replaced with a random sample. (Transient failure is a separate, faster path — a peer that blows response timeouts under a long GC or under-provisioning is dropped immediately, not slowly churned.) This design is largely eclipse-resistant rather than a ratchet: the resample is random, so a Sybil fleet gets no better-than-proportional odds of backfilling a freed slot, and the worst-20% eviction is absolute, so it also culls the weakest of the adversary's own peers — even a brief local eclipse leaks back out as those marginal peers churn and random (mostly honest) draws replace them. What survives is only the patient top-scorer of the reputation-farming attack: an adversary can farm score to hold a hot slot, but it cannot use the churn machinery to displace honest peers or take over the set. Cheaply forcing an honest peer out isn't available — only expensive, on-path degradation (nation-state / ISP-tier, and not ranking-specific) can. Diversity-/anti-eclipse-aware selection (Improvement #8) hardens the residual, but the base mechanism already does most of the work.

Handling advertised-size manipulation (skeleton)

The estimate's size input is advertised, hence gameable (intro). The organising fact is that for a hash-named datum the content fixes the true size, so any disagreement is a provable lie once the datum is in hand — the difficulty is only the choice before the download.

  • All advertisers agree. No steering — the common size scales every estimate identically, so ranking still turns on speed; run it normally. The twist is what it means if that agreed size proves a lie on download: no reachable peer told the truth, so either the whole neighbourhood is dishonest or we are eclipsed. That is worth acting on beyond demoting the advertisers — but the response (defensively resampling for fresh peers to break a captured view) belongs to the peer-selection / anti-eclipse layer (Improvement #8), not to the delivery score. (A uniform honest size is entirely benign, misinforming only absolute deadline-feasibility, not the peer choice.)
  • Genuinely distinct datums (different hashes — several VRF election winners). Not equivocation; a capacity / protocol-storm question rather than a size dispute. Fetch what you can, prioritising by the consensus signal — which candidates the votes favour — and estimate each independently.
  • Same hash, conflicting sizes — the weaponised case. Because the hash fixes the content, a divergence is at least one peer lying.
    • Neutralise the incentive. Do not let the smallest advertisement win the estimate — that is the lever the attacker pulls. Feed a robust aggregate size (the plurality value across advertisers, or conservatively the max) into every peer's estimate, so understating buys no ranking advantage and the choice falls back to measured speed — i.e. prefer the higher-ranked peer, with a random tiebreak among equals.
    • Punish after the fact. On delivery the true size is known; a peer whose advertisement was inconsistent with it is provably lying and is demoted — a stronger, cleaner form of tx-submission's deviation penalty (inconsistency with canonical content, not measurement noise).
  • Conflict arrives mid-fetch (a second peer offers the same datum at a different size while the first is in flight). Same recourse as a slow peer — wait, reset, or hedge — but the switch must be driven by the first peer's observed progress, never the newcomer's claimed smaller size; hedging on the claim alone rewards the lie in dynamic form.

Two structural notes bound the problem. If the size is committed in signed metadata — an EB header or certificate — the advertisement becomes verifiable a priori and the lie is impossible, closing the hole for EBs and leaving only unattested aggregates such as closures. And the damage is bounded regardless: the worst outcome of a size lie is a single recoverable mis-selection — caught by actual-vs-predicted, remedied by hedge/reset — never a failed fetch.

The window-sizing tension, and the gated extension that resolves it

Requests are small — often a single SDU — but responses arrive as a burst of many SDUs, and the cookie on our request is echoed back on that burst. The first response SDU's echo gives the RTT sample; the follow-up SDUs of the same burst feed estBurstS, the serialisation rate. That poses the question the burst window exists to answer: for how long do we keep attributing incoming SDUs to this response's burst — counting them toward its throughput — before a new SDU counts as a fresh exchange instead? That horizon is rttBurstMaxAge (hard-coded 1 s today).

It has to be two things at once: tight enough that a pacer can't stall-then-look-fast, and extensible enough to measure a genuine multi-second closure. A fixed 1 s is a poor compromise — too short for a large honest transfer, too generous to a pacer if loosened. And the extension is not merely about finishing the measurement: a longer sustained burst accrues the byte-coverage that earns an estBurstS its confidence weight (see the appendix), so closing the window early yields not just a shorter estimate but a less trustworthy one.

When a burst would otherwise time out mid-transfer there are two choices: stop counting (and leave honest data on the table if the peer was good and the stream healthy), or keep extending the deadline while the peer is still actively serving. The second is the gate — extend while the peer serves promptly, end the window when it goes quiet.

"Serving promptly" can be read off network-mux's cookie mechanism (a branch feature today, not yet on main): we piggyback cookies on our outbound SDUs, the peer echoes them back, and the still-unechoed cookies sit in a queue — the cookie PSQ — whose matched echoes yield the RTT samples. Repurposing that queue as a liveness gauge is the new part: a peer echoing promptly keeps its oldest outstanding cookie young; a peer gone quiet lets the oldest cookie age and the queue grow. So there are two readings of "serving promptly":

  • Cookie echoes (general): extend while the cookie queue isn't growing / its oldest cookie isn't aging.
  • Incoming SDU spacing (one-directional download, where our cookie-carrying outbound is sparse): extend while successive SDUs arrive within a tight back-to-back max-spacing.

Either way an honest large closure keeps the window open (its SDUs / echoes keep coming) while a pacer that goes silent to game the estimate forfeits the extension. The tight spacing has one honest hazard — a loss / RTO briefly stalls the in-order stream and can trip it — so the anti-gaming does not rest on the gate alone: it rests on the actual-vs-predicted backstop (below), which tolerates a one-off trip and churns only a peer whose delivery is consistently worse than its advertised rate.

Using the RTT signals — and a circularity to avoid

The queue is a liveness gate on the burst extension (above): a trend — is the peer serving now — read from whether the sent-cookie queue is growing / its oldest cookie is aging. It is deliberately not a comparison of queue duration against the self-measured RTT quantile: that would be circular — an adversary delaying echoes uniformly inflates the queue front-age and the measured RTT together (a cookie echoed at t0 + RTT + D gives both a front-age peak and an RTT sample of RTT + D), so the comparison never trips. The trend has no such weakness: a silent peer's queue grows whatever its RTT was.

The self-measured RTT is used heavily — as an input, not against itself. The idle-gap gamer's inflated RTT feeds the estimate and self-penalises: high apparent RTT ⇒ high time-to-deliver ⇒ ranked down, no independent reference needed (the bind and bigger-picture sections, below). Busy-fill — keeping the burst alive with filler SDUs rather than going silent — is caught the other way: the filler is counted, so estBurstS reads the true, slow rate and the estimate ranks the peer down. (Filling fast with non-closure bytes to hold estBurstS low instead is a different attack, byte-padding — the one escape these self-measured signals miss, caught by the per-fetch check; see the bind.) So the scheme deprioritises a gamer on self-measured signals alone.

An independent RTT reference buys cleaner attribution, not the core defence. A kernel tcpi_rtt or handshake baseline gives a single-observation tell of adversarial app-layer echo-holding (kernel RTT low, app-RTT high ⇒ the peer is sitting on cookies — literally the existing "app-RTT vs kernel-RTT divergence" cross-check), distinguishing it from an honestly far peer (both high). That matters only for escalating from deprioritise — which self-measured signals already do — to demote/blocklist.

Attribution isn't gated on it, though: the actual-vs-predicted comparison (below) recovers a form of it portably — a persistent actual ≫ predicted identifies targeted withholding using only app-layer signals (direct T_wait, estBurstS, RTT), no kernel/handshake reference. It convicts more slowly (it needs persistence to clear the serve_time/cold-cache confound) and it catches the attack by a different tell.

The two attacks leak differently — idle-gap through RTT, byte-padding through the payload — as the bind section below sets out. So the independent reference's portability caveats (Linux + tracetcpinfo, design-phase, for tcpi_rtt; drift-prone for the handshake baseline) cost only the fast, clean attribution signal — not attribution outright, and certainly not the core defence.

The adversary bind — and why aggregate estBurstS is a strength

What makes the gate sound is a genuine dilemma the adversary faces on a single measured stream:

  • To make delivery slow, the stream must be idle a lot (low byte-rate).
  • To game a gap-based estBurstS, that idle must fall outside bursts (the burst timer expires during silence, so the stall isn't counted).
  • But idle = no SDUs = no echoes → the cookie queue ages / RTT samples inflate.
  • To keep echoes prompt, it must fill the idle with SDUs — but then the burst stays alive, the fill is folded into estBurstS, and the estimate reads the true (slow) rate.

So it cannot simultaneously game a gap-based throughput estimate and keep the liveness/echo signal flowing: whichever activity carries the prompt echoes is measured, and going silent to hide the stall shows up as echo lag. Pick one, caught by the other.

Cardano's mini-protocols are mostly pull-based, which tightens this further: the peer sends SDUs only in response to our requests, so it cannot inject filler at will to keep echoes flowing. The fill would have to be responses to some other request we happen to have outstanding to it — an opportunity it cannot manufacture — so busy-fill is opportunistic and hard to exploit reliably. Absent such traffic, the only way to withhold our closure is to fall silent, which the echo-lag / SDU-spacing gate catches directly.

Aggregate (protocol-mixed) estBurstS is what closes the loophole. The bind holds only if the echo-carrying SDUs are inside the measurement — and today estBurstS is connection-level, so every received SDU, whatever mini-protocol carries the echo, feeds the aggregate burst. There is no unmeasured side-channel to park prompt echoes on. The protocol-mixed aggregate we usually treat as a limitation is, for this attack, the strength.

Caveat for a future per-protocol split. The loophole reopens if per-protocol estBurstS (the proposed "Shape B" split — per-mini-protocol DeltaQ buckets, not implemented today) is added without also binding the cookie/RTT per-protocol — because then the throughput signal and the echo signal live at different granularities. Concretely: we fetch a 12 MB closure over LeiosFetch (protocol B) and are also driving a second mini-protocol A (keep-alives, say, or another fetch) on the same connection.

  • It delivers the closure in tight 50 kB micro-bursts, then goes silent on B for ~500 ms, repeat. Per-protocol estBurstS_B counts only the fast micro-bursts — the 500 ms silences exceed B's burst gap and fall outside its bursts — so it reads fast. Gamed.
  • During each B-silence the peer keeps answering our A requests promptly, each response echoing our latest cookie. The cookie PSQ is connection-level, so those A echoes keep the queue drained and RTT low — the gate stays quiet.

Result: estBurstS_B says LeiosFetch is fast and the gate says nothing is held back → we rank the peer a great closure-deliverer, route closures to it, and it dribbles them. A is a side-channel that estBurstS_B doesn't see, yet it still satisfies the connection-level gate — exactly the gap the bind depends on being closed. The loophole is gated on A traffic frequent enough to keep draining the connection-level queue through the B-silences, though — consistent with the pull-based limit above, the peer can't fabricate that; it can only ride A traffic we ourselves drive.

The aggregate estimate doesn't have this gap: those same A responses are inside the aggregate burst, so they either keep it alive (⇒ estBurstS reads the slow closure rate) or, if the peer goes fully silent to avoid that, the echoes lag (⇒ caught). The fix under a per-protocol split is to keep the two signals at the same granularity — judge B's delivery against echoes that ride B, so a silence on B costs echo-lag on B. Rule: go per-protocol on estBurstS ⇒ go per-protocol on the echo/RTT too.

Where the payoff hides — and how each strategy leaks. The adversary's payoff is always the same: pace the bursts so estBurstS reads fast, win the fetch, then dribble the payload. Stalling echoes or inflating RTT is never the goal — it is only a leak of the pacing, and a different signal catches each leak.

  • Busy-fill — fill the stalls with SDUs so the echoes stay prompt and nothing looks held back. But those filler SDUs keep the burst alive, so they fold into estBurstS, which then reads the honest, slow rate. estBurstS itself catches it.
  • Idle-gap — go silent between micro-bursts so the stalls fall outside the burst and estBurstS reads fast. That same silence starves the echoes: the cookie front-age grows and the RTT-quantile inflates. RTT catches it — self-penalising in the ranking, and, against an independent reference (kernel tcpi_rtt / handshake), attributable as app-layer echo-holding.

(The queue-vs-self-RTT comparison stays circular — uniform echo delay inflates the front-age and the RTT sample together — but the bind means we never need it.) The third leak, byte-padding, escapes both of these and is treated next.

The one escape the bind does not close: byte-padding. The peer sends other bytes fast, serving another protocol request, so the aggregate byte-rate looks high while it dribbles the closure you actually asked for. estBurstS reads fast, the padding carries prompt echoes (the gate stays quiet), yet your closure is slow. A measurement bound to that specific fetch — direct completion timing (T_wait on the closure; next section) — catches it, because the padding isn't the bytes you're waiting on. And comparing that T_wait to the peer's own estimate upgrades detection to attribution: under padding the prediction is fast but the actual is slow, and actual ≫ predicted means the peer has the capacity (it's spending it elsewhere) and is withholding from us (the data-withholding threats T20/T21), not merely slow. Lose-lose for the padder: advertise fast and dribble ours ⇒ discrepancy; advertise slow to match ⇒ ranked down anyway; deliver ours fast ⇒ not the attack. Caveat: honest confounds fire the same discrepancy — server-side serve time for our closure, HOL/contention on the shared bearer, stale estBurstS/RTT — so deprioritise on a one-off and punish (demote/blocklist) only on persistence with those ruled out.

The serve-time confound is the sharpest, because it produces the same observable, not just a noisier one. estBurstS/RTT capture the byte-flow rate and the network round-trip, but not the peer's cost to produce the bytes — reading our closure's txs from its store, reassembling it:

actual    = RTT + serve_time + size × per-fetch-rate + loss/RTO
predicted = RTT +              size × estBurstS       (aggregate signals)

The prediction has no serve_time term, so actual − predicted ≈ serve_time: a large serve time alone reads as actual ≫ predicted. So a peer whose TxCache holds others' closures (hot, served fast) but not ours (cold, disk-read) shows a fast aggregate and a slow closure — capacity that exists but isn't reaching our request, exactly like byte-padding, yet benign (which closures are hot/cold is diffusion history, not the peer choosing to starve us). Decomposing our fetch into time-to-first-byte (which absorbs serve time) vs the whole-fetch byte-rate separates the two if the peer batch-reads (long TTFB, then a normal rate ≠ dribbling) — but a slow streaming read shows a slow rate throughout and is indistinguishable from withholding by rate alone. So the discrepancy stays a deprioritise signal (correct either way — the peer is slow for us now); promoting it to malice needs the slowness to be systematic across our requests to that peer (targeting), not per-request random (cache-luck).

The queue is the structure — the extension gate

The cookie/promptness queue is the load-bearing structure: the gate that lets the burst window extend to measure an honest large closure while denying a pacer the same extension (the window-sizing tension above). The consistency and actual-vs-predicted checks sit on top of it as the backstop, not as a substitute for it. It reuses machinery already present — the t-digest and the cookie PSQ; what is new is gating the extension on them.

Setting the threshold for the actual-vs-predicted check. For the byte-padding check (how much may actual exceed predicted before we act), don't derive the band from estDeltaQVVar. It's the wrong granularity — per-SDU micro-jitter, which ignores the correlated per-fetch terms (ramp-round count, loss-recovery, serve_time), so it under-estimates the real spread and the band comes out far too tight — and the wrong shape, since the residual is heavy-tailed and a σ-band mis-fits it. Threshold instead against an empirical high quantile (say p99) of the residual actual − predicted, from a population baseline across honest peers — not the suspect's own samples, which it can inflate to widen its band and hide under.

Stratify that baseline by regime, because the residual's spread is not one distribution. It scales with message size (more bytes ⇒ more loss-recovery exposure and a longer transfer) and with RTT (both the ramp-round count and the RTO magnitude grow with it — and that dependence survives even though predicted already subtracts a ramp(RTT) term). Loss rate is the natural third axis, but it is non-portable — only via kernel tcpi_retrans/tcpi_lost (Linux + tracetcpinfo, design-phase) — so a first pass stratifies on size × RTT only and pools clean and lossy paths in the same bin. The cost is a deliberately conservative band: the lossy fetches in a bin inflate its p99, widening the threshold, so first-pass detection is less sensitive (a padder can hide under the loss-inflated band) but rarely false-positives. Add the loss stratum to tighten it once kernel telemetry is available on the measuring node.

The bigger picture: rank on measured completion, not estBurstS

The pacing attack only matters to the extent we use estBurstS as a proxy for delivery quality (peer ranking, chunk sizing). But those decisions are moving onto direct completion timing (T_wait / observed arrival). Against direct completion timing, pacing is self-defeating: the adversary's paced-slow delivery is the measured completion time, so it ranks low no matter how it games the per-SDU gaps. So the highest-leverage defence against estBurstS-pacing is not to trust estBurstS for the decision that pacing targets — rank on observed completion, and pacing buys the adversary nothing.

Even short of direct timing, ranking on the estimate (Improvement #1's fused metric) already absorbs the idle-gap gamer, which self-penalises: faking a low estBurstS by going silent forces its apparent RTT up (held cookies), and the RTT term then dominates its estimate, so a genuine low-RTT/high-throughput peer always out-ranks it (higher apparent RTT ⇒ higher time-to-deliver). This needs no independent cross-check — you never have to distinguish the gamer from a genuinely far peer, you just rank both down for their RTT; the tcpi_rtt/handshake anchor is only for attributing and punishing the gamer specifically. Two caveats: the ranking RTT-quantile must be high enough to capture the stall inflation (occasional long stalls hide above p80), and byte-padding evades it — it shows low RTT (see the bind). So the ordering is: measured completion timing closes both; the fused estimate is a good fallback that handles idle-gap for free.

That leaves estBurstS needing hardening only for its residual uses — diagnostic decomposition, and slow-loris detection. Slow-loris is the mirror image of pacing: the peer makes no attempt to hide its slowness, it just accepts the fetch and trickles it, tying up our slot. Because it looks slow, the time-to-deliver ranking deprioritises it for free; and a peer whose estBurstS rises mid-transfer — throughput collapsing below a floor — is the slow-loris signature (Improvement #3). Answering it is harder than spotting it, though: there is no per-fetch cancellation today, so the current stuck fetch has only three imperfect moves — wait (risking the deadline), reset the connection (degrading our peer topology and, done reflexively, opening an eclipse surface — and a slow peer can beat the even-slower one we might draw to replace it), or hedge to another peer (Improvement #5), the most resource-intensive, since our socket buffers and ingress must then service both the slow-loris and the hedge. The deprioritisation above is the only lever needing no cancellation — but it is preventive (we stop choosing the peer), not curative (it can't rescue a fetch already in flight). So estBurstS earns its keep here in the opposite direction from pacing — flagging a peer that looks slow, not one faking fast — where pacing-to-look-fast can't help the adversary. The estBurstS-vs-RTT consistency check, meanwhile, is worth keeping as a sanity gate for when you do lean on estBurstS, but it shouldn't be load-bearing.

Why leaning on estBurstS is sound

estBurstS is low-level and gameable, so building the prediction on it looks risky — but it's safe because of the role it plays:

  • Center-line, not tolerance. It supplies the peer's advertised rate in the estimate. The tolerance — how far actual may exceed that — is a separate quantity: a high quantile (p99) of the residual actual − predicted measured across honest peers (the threshold discussion above). Call that honest-peer residual distribution the baseline. estBurstS's missing macro terms (serve_time, stalls, loss-recovery) don't corrupt the check: because predicted omits them, they surface in honest peers' residuals too, so they're already folded into the baseline — a suspect peer is flagged only by deviating above it.
  • Audited, not trusted. actual (direct T_wait) is the auditor; predicted (from estBurstS) is the claim under audit — its unreliability is fine because it's the thing being checked, not the ground truth.
  • Inaccuracy cancels; gaming self-defeats. Because that baseline is computed with the same estBurstS, any systematic bias in estBurstS (say it reads 10% fast for everyone) shifts every peer's predicted alike, shifts the baseline with it, and cancels in the residual-vs-baseline comparison — so the check needs estBurstS consistent across peers, not accurate. And the only useful lie is to make estBurstS look low, which shrinks predicted and makes the check more sensitive; lying it high advertises slow and ranks the peer out.

The one role estBurstS is not given — trusted ground truth for ranking — is the one it couldn't survive; that's why ranking ultimately prefers measured time-to-deliver. Correspondingly, the tolerance band is the empirical residual, not the per-SDU estDeltaQVVar (the threshold discussion above).

Smaller points

  • Your own GC is a confound for the gate. A GC pause makes you timestamp echoes (and incoming SDUs) late, inflating the cookie front-age and the SDU-spacing reading — looking like the peer stalled, a false extension-revocation that blames the peer for your own pause. Same class of false trip as a loss/RTO on the gate (the window-sizing section); the backstop's persistence requirement absorbs it. Kernel tcpi_rtt is immune to your GC — a reason to prefer it for the RTT input where available.
  • Lazy sampling. The RTT quantile feeding ranking/prediction is read from the t-digest every ~1 s in processIngress, not per-SDU — preserving the finger-tree's deferred <>. (The extension gate itself is a cheap front-age / SDU-spacing check, separate from the t-digest.)
  • Not tuning-free. The knobs are the promptness threshold (the tight back-to-back SDU spacing / front-age), the backstop quantile (p99), and the churn-persistence count. The tight threshold's risk is false trips (loss/RTO, GC), not gaming — a steady-dribbler that stays just under it keeps the burst extending and is then measured slow, which is correct — and those false trips are what the backstop's persistence tolerates. The knobs are physically grounded (a spacing in milliseconds, an RTT, a percentile), but they still need calibration.
  • Download liveness needs no cookies. In a one-directional download the incoming-SDU-spacing gate supplies liveness directly; the cookie signal (for RTT) can go stale there — cookie-carrying outbound is sparse — so inject a cookie periodically if you need a fresh RTT through a long download.
  • How the signals relate. We compute estBurstS ourselves from the peer's SDU gaps, so a peer influences it only by pacing its own SDUs; the cookie echoes are an orthogonal measurement we control. Pacing the SDUs and starving the echoes coincide only when echoes ride the measured stream — which aggregate estBurstS guarantees (the bind section); a per-protocol split would have to preserve it.
  • Response on detection is cheap and low-blast-radius: the ranking already deprioritises a gamer; on a persistent actual-vs-predicted discrepancy, churn/demote it. It's a peer trying to look good, so a false demotion costs little.

Bottom line

The window-sizing tension — tight enough to deny a pacer, extensible enough to measure a large closure — is resolved by gating the burst extension on peer promptness (cookie-echo front-age, or incoming SDU spacing on a download): an honest large transfer keeps the window open, a pacer's silence closes it. The gate is imperfect alone — a loss/RTO can trip the tight spacing, and steady-dribble or byte-padding slip through — so the anti-gaming rests on the actual-vs-predicted backstop: churn a peer whose delivery is consistently worse than its advertised rate, which also absorbs the gate's false trips via the consistency requirement. An independent RTT reference (kernel tcpi_rtt / handshake) is a nice-to-have for fast attribution, not a prerequisite. And all of this hardens the prediction: the ranking itself prefers directly-measured completion wherever a fetch has finished, where a pacer gains nothing.

Appendix: attributing confidence to estBurstS

An estBurstS built from a long, sustained transfer — one where the burst window kept extending across many SDUs — deserves more weight than one scraped from a short or paced burst. To lean on the good estimates and discount the thin ones, estBurstS needs a confidence attached to it. Two candidate measures suggest themselves — the variance of the estBurstS regression, and the number of samples it was built from — and the useful answer is that count is the right instinct, but for a reason that is structural rather than statistical.

Variance and count are not rivals. If estBurstS is a fitted slope (or a mean rate over near-constant SDU sizes), the statistically correct confidence is not the raw residual variance — that is estDeltaQVVar — but the standard error of the estimate, SE ≈ residual_sd / √N, which already folds the residual variance and the sample count into a single number. So the choice between "use the variance" and "count the samples" is a false one: SE is their principled combination.

But variance — and SE with it — is the wrong signal for telling an honest estimate from a paced one, and is in fact perverse. A metronomic pacer emitting perfectly regular micro-bursts produces near-zero residual variance, hence a tiny SE, and would therefore score as high confidence. Variance measures precision and consistency, not honesty or representativeness; a precise measurement of a gamed rate is still gamed. Treating low variance as a positive trust signal rewards exactly the regular emission a pacer can manufacture. (This is the same reason estDeltaQVVar is rejected for the tolerance band above — it measures the wrong thing.)

Byte-coverage is the trustworthy quality signal, for a structural reason. Counting is the right instinct, with two refinements. First, weight by bytes, not raw SDU count: a flood of tiny SDUs runs an SDU counter up cheaply, whereas accumulated payload (Σsize, ideally as a fraction of the transfer) cannot be inflated without actually delivering bytes. Second — and this is the crux — coverage is trustworthy not because a large N is statistically comforting but because it is downstream of the extension gate. Under the gate, coverage can only accumulate by serving promptly for the whole burst, and serving promptly throughout is precisely what forces estBurstS to the true sustained rate (the bind). A pacer that stalls forfeits the extension, so its burst stays short and its coverage low. High byte-coverage therefore certifies a gate-vetted, sustained, prompt-served transfer — and hence an honest estBurstS, whether that honest rate is fast or slow. The coverage is less a confidence-in-a-number than a receipt that the gate let the estimate run long, which it does only for honest service; it costs one accumulator.

The two measures then do different jobs. Byte-coverage is the trust weight: a thin, low-coverage estimate is shrunk toward a prior — the peer's long-run estBurstS, or a population rate — in the manner of empirical-Bayes shrinkage, while a high-coverage estimate is trusted as measured. This is how a long extended burst earns its keep — it accrues weight — and it handles the cold-start case gracefully. SE, meanwhile, is the error bar on the estimate itself — the uncertainty in estBurstS, and so in the predicted time-to-deliver — kept distinct from the tolerance band, which stays the empirical residual across honest peers (the threshold discussion above). Its role is to hold the actual-vs-predicted check back when a discrepancy lies within the prediction's own error bars — a thin estimate shouldn't convict — not to set the detection threshold.

None of this is the anti-pacing mechanism: the gate and the actual-vs-predicted backstop are what catch a pacer. Confidence-weighting only decides how much to lean on an estBurstS already vetted, and how hard to shrink a thin one toward the prior.

Clone this wiki locally