Skip to content

leios testnet

Jump to bottom
Marcin Wójtowicz edited this page Jul 13, 2026 · 8 revisions

DeltaQ report validation and driving Leios parameters

Testnet trace collection

Leveraging DeltaQ tracing mechanism (including the proposed TraceRecvBurstSDU):

  • Aggregated TraceRecvDeltaQSample events (one per peer per protocol per ~10 s) is cheap: a ~100-node testnet with ~60 peers each gives you ~600 samples/sec cluster-wide, order-of-megabytes/day. Trivial for InfluxDB/Timescale/Prometheus.
  • Raw TraceRecvDeltaQObservation and TraceRecvBurstSDU events are far higher volume (per-SDU). Ship them selectively — only enable for a subset of nodes, or during measurement windows — and land them in S3 as Parquet for offline analysis. This is the pattern the mempool-measurements dataset already uses.
  • PeerRTT.quantile is a live STM reader — you can either poll it periodically at whatever cadence you want, or (better) emit it as a periodic aggregate on the same 10 s cycle.
  • TraceTCPInfo is already wired in the mux bearer, but conditionally: it requires Linux and the tracetcpinfo cabal flag on network-mux at compile time (default off; flag description flags it as "costly"). When enabled, per-SDU it emits kernel state (tcpi_retrans, tcpi_lost, tcpi_snd_cwnd, tcpi_advmss, tcpi_rcv_space) — the primary loss-rate and rwnd answer for the validation frame. In practice: compile a dedicated tracetcpinfo build for the Linux measurement nodes only, run the vanilla build everywhere else. Treat the resulting numbers as testnet / design-phase telemetry only, not runtime signals — the flag is off in production builds by design (see leios-overlap.md's Frame 1 subsection on TCPInfo for the design-phase framing). Additional motivation: tcpi_rtt cross-checked against the branch's PeerRTT.quantile (application layer) is one of the multi-signal-fusion cross-checks in leios-fetch-scheme.md — a peer whose application-layer RTT is much larger than the kernel's TCP-layer RTT is a signature of application-level adversarial delay.

So the infrastructure question reduces to "hook initDeltaQTracer' up to cardano-tracer and add a Grafana dashboard." References to Frame's target the leios-overlap.md report, which discusses expanded ouroboros-network tracing machinery to validate concerns in the DeltaQ report and which in turn is used to analyze Leios itself.

Validation vs. parameter-driving — different use cases

Validation needs known ground-truth inputs and measured outputs. For eg. to verity the Mathis relation in the deltaq report, measure estBurstS and p, which requires either the TraceTCPInfo kernel telemetry already emitted by the mux (Linux + tracetcpinfo cabal flag; see the trace-collection section above and leios-overlap.md's Frame 1 subsection on TCPInfo) or the S-residual probe (a noisy secondary derivation). leios-overlap.md recommends both a controlled tc netem microbench (where you dial p and RTT explicitly) and a testnet campaign for different purposes:

  • Microbench: does the model form (Mathis or CUBIC) match observed throughput at known p, validating the equation.
  • Testnet: what is the empirical distribution of estBurstS, RTT quantiles, per-protocol tx-arrival latency across a realistic cross-continental mesh? Feeds the inputs to the equation and tells you whether the CIP-0164 assumed values are conservative or optimistic.

Parameter driving (what should CIP-0164 say?) is exactly what a testnet gives you well. Concretely, the observations feed into the following levers, grouped by the same three tiers used in leios-overlap.md's Frame 1 subsection:

Leios parameter Data you'd use Tier
S_EB_tx maximum Largest closure size such that the 7 s voter deadline is met at p95 with observed estBurstS Protocol parameter
L_vote (currently 4 s, tunable component of the 7 s voter deadline) p95/p99 of voter validate CDF — 3·L_hdr + L_vote + observed estBurstS·closure_size + µ_eff·N_txs Protocol parameter
L_diff (certRB diffusion budget, currently 7 s) p95/p99 of blended-multi-hop estBurstS × 8 kB (certRB body) Protocol parameter
Round length (3·L_hdr + L_vote + L_diff, currently 14 s) Sum of the two above, plus header diffusion time Protocol parameter
p (loss-rate assumption in report.md §4.2, currently 10⁻⁴) TraceTCPInfo tcpi_retrans / tcpi_lost distribution across connections Rationale-doc assumption
Mathis-vs-CUBIC choice (report.md:1046-1173) TraceTCPInfo tcpi_snd_cwnd trace picks the actual growth law Rationale-doc assumption
long_hop_owd (currently 134 ms) Empirical p95 of PeerRTT binned by peer geography Rationale-doc assumption
rwnd-vs-cwnd binding (§5.4 caveat 1) min(tcpi_snd_cwnd, tcpi_rcv_space) reveals which is the actual limit Rationale-doc assumption
BlockFetch per-peer byte watermarks (calculatePeerFetchInFlightLimits) Empirical PeerGSV from the branch's window; cross-check via tcpi_delivery_rate × tcpi_rtt on Linux Node-implementation constant
SPO tuning guidance (MSS, IW10, congestion-control choice) TraceTCPInfo per-connection deltas from expected defaults Node-implementation constant (ops guidance, not spec)

TCPInfo-fed rows are Linux-only, testnet / design-phase only (see the trace-collection section above). All other rows are portable and can be collected on any mux bearer.

For the runtime-adaptive lever the branch also enables (a voter refusing to vote when its local mesh health drops below CIP-0164's assumed budget), see the reader-view section below — it belongs in Frame 2, not in this design-time parameter-driving table.

What a good testnet campaign looks like

For the Leios use case specifically:

  1. Topology: 30–100 nodes across ap-northeast-1, eu-central-1, us-east-2 (mirroring the mempool-measurements setup so π₁ and network data live in the same dataset), plus 5–10 "adversarial edge" nodes on non-AWS or consumer-grade uplinks to probe §7 limitation 6 (South America ↔ APAC, deep-buffer bottleneck links). A pure-AWS mesh misses the tail (higher loss, lower bandwidth, etc.) that actually determines feasibility, ie. successful Leios rounds.

  2. Baseline traffic: run Praos with mainnet-scale block production, plus synthetic tx-submission load matching mainnet mempool arrival rates. The Leios pipeline itself may or may not be running; if it is, you're measuring end-to-end; if it isn't, you're measuring the underlying transport under representative load.

  3. Instrumentation: TraceRecvDeltaQSample on every peer for aggregates. Raw TraceRecvDeltaQObservation on 3–5 designated "measurement nodes" (one per region) for high-resolution offline analysis.

  4. Duration: a week, per §5 of the mempool-measurements dataset. Any shorter and you miss diurnal loss patterns — the day-night cycle of network usage, where packet loss on the same path can swing by orders of magnitude between low-traffic hours (~1e-5) and evening peak hours (~1e-3). A short measurement biases the observed p toward whichever part of the cycle you sampled; a week captures the full cycle plus weekend-vs-weekday variation, giving stable tail estimates. Directly plottable from tcpi_retrans against local time-of-day.

  5. Controlled perturbations: on a sub-mesh, inject tc netem shaping to force known p, RTT, and loss patterns. This gives you the ground-truth pairs needed to falsify Mathis vs CUBIC (i.e., Experiment 1 from leios-overlap.md, run inside the testnet rather than on a two-node isolated rig).

  6. LeiosFetch-scheme validation (if a candidate scheme is deployed on the testnet). Compare the four configurations from leios-fetch-scheme.md's suggested experimental design: memo's scheme as written (baseline), memo + Improvement #1 (hybrid classification only), memo + Improvements #1–#8 (full adaptive variant with fusion and diversity-aware churn), and the erasure-coded variant (Alternative B, if prototyped). Metrics per configuration: empirical $F_{\text{full}\mid C}(14)$, P99 completion time, redundant egress per round, and adversarial resistance under the injection scenarios in item 7 below.

  7. Adversarial injection: designated adversarial nodes on the sub-mesh with controlled attack profiles. Cheap to add (mostly tc netem on the designated nodes) and tests defensive mechanisms that would otherwise be untested until real-world adversaries appear:

    • Slow-loris: adversarial node accepts fetch requests then delays response indefinitely. Tests Improvement #3 (estBurstS-based detection) and its pacing-attack variant (delivering just above the detection threshold).
    • RTT inflation: adversarial node adds artificial delay to all responses. Tests the multi-signal fusion cross-checks (PeerRTT.quantile vs tcpi_rtt divergence, geographic minimum RTT). Both are named in leios-fetch-scheme.md's fusion section.
    • Chunk-hedge slow-loris: adversary is one of many NBL peers each getting a chunk; it slow-lorises its assigned chunk. Directly tests the memo's key adversarial concern about block-completion under partial-peer failure.
    • Topology-induced peer concentration (non-adversarial but shares the measurement path): configure a sub-mesh with a stake-concentrated "dense" region and a sparse region with only 2–3 high-stake SPOs. Measure whether churn drives mesh-wide big-ledger peer sets toward the dense region (as predicted by the per-block-origin model in leios-fetch-scheme.md's Improvement #8) and whether sparse-origin blocks then see low mesh-wide seed-hop coverage. Compare against a diversity-aware-churn deployment to quantify how much the mitigation buys.

    Sub-mesh scale for adversarial injection should mirror the real-world adversary size assumption (e.g., ~4% of the big-ledger pool, per the arithmetic in leios-fetch-scheme.md's seed-node section).

Where this connects to work already in flight

The mempool-measurements dataset that fed pi1_derivation.md is already collecting some of what you'd need — per-node tx-arrival timestamps. Extending that pipeline to also collect this branch's DeltaQ traces gives you π₁ and the network side from the same nodes in the same time window, which lets you cross-validate: does a period of high π₁ correlate with degraded estBurstS (i.e., is mempool divergence caused by network stress)? That's an interesting question the report can't answer today.

Similarly, the IOG benchmarking cluster that runs network-benchmark topologies is a natural fit — it's already designed to run cross-continental Cardano nodes under controlled load.

The short answer

Yes, a real cross-continental testnet is the right validation vehicle, especially for parameter-driving Leios. The traces are already collectible via the existing cardano-tracer infrastructure. Combining testnet measurement with a controlled tc netem microbench gives you both the inputs (empirical RTT/S distributions on real paths) and the falsification (does Mathis or CUBIC fit at known p) that the report currently lacks. The main gaps are (a) adversarial-path coverage — pure-AWS meshes under-represent the tail, and (b) the MiniProtocolNum enrichment needing to be on the deployed nodes if you want to test the 1-hop pre-diffusion assumption. Neither is a fundamental obstacle.

That covers the validation-and-parameter-driving frame. The rest of this document covers three topics that don't fit inside that frame:

  • Production cost of the DeltaQ machinery — running the branch's RTT/DeltaQ machinery on public relays (with hundreds of inbound peers) has real cost. What that cost is, what mitigations exist, and how they interact with the two Leios frames.
  • Adaptive Leios via the PeerRTT reader view — the branch enables Leios mini-protocols to observe their own network conditions at runtime and adapt behaviour, rather than baking assumed conditions into CIP-0164 forever. This is the reader-view expansion below.
  • Fully-blended diffusion when 1-hop fails — what happens, how to diagnose it, and how DeltaQ traces expose dynamics the report itself doesn't model. Covered further down.

Production cost of the DeltaQ machinery

Existing Cardano practice keeps DeltaQ analysis strictly on the trace path (rather than in-band with protocol logic) precisely because it's potentially expensive, and BlockFetch derives its PeerGSV from KeepAlive-driven samples (once every ~10 s) rather than a richer signal for the same reason. This part covers what the branch adds on top of that existing cost, when it matters, and how to mitigate.

What the branch adds on top of main's existing cost

Main already emits TraceRecvDeltaQObservation per SDU and runs TraceStats.step on every observation. That cost is proportional to SDU rate across all connections. The branch layers on:

  • Cookie mint (egress). PRNG draw + OrdPSQ insert per SDU, amortised by 1 ms mint-batching to roughly "one cookie per outbound burst" rather than per SDU.
  • Cookie match (ingress). PSQ.atMostView per received SDU: matched-echo lookup + age-out of expired cookies in one operation. O(log n) where n = outstanding sends.
  • Per-protocol StatsA lookup. Map MiniProtocolNum around the existing per-size record structure. Constant-factor increase (~5x for 5 active mini-protocols).
  • Burst tracker. Single-slot cursor per connection, essentially free.
  • PeerRTT t-digest update. O(log δ) per matched echo, δ = 100.
  • TVar write for the reader view: one STM commit per matched echo.

The per-SDU addition is roughly a doubling of main's existing DeltaQ work. Storage per connection: a few kB for the t-digest, a few kB for the per-protocol StatsA state, plus the cookie PSQ (capped at a few thousand outstanding × ~8 B ≈ tens of kB).

Rough arithmetic for a heavily-loaded public relay (500+ connections, ~10 k SDUs/s aggregate, ~5–10 µs of DeltaQ work per SDU): 5–15% of a core and 15–30 MB of cookie PSQ storage. Not catastrophic in isolation, but additive with everything else on the node — and the kind of overhead current Cardano practice explicitly refuses to pay in the block-fetch decision path.

The tx-submission direction inversion

Almost every Cardano mini-protocol carries data from responder to initiator — block-fetch, chain-sync, keep-alive. Tx-submission is inverted: the initiator sends transactions to the responder. See track.md's cross-cutting note on this for the general framing.

Practical consequence for cost mitigation: from a relay's perspective, the tx-submission traffic we care about for pre-diffusion monitoring arrives on inbound connections — where this node is the responder, and where peer count is highest (tens to hundreds). Blocks arrive on outbound connections, which is the low-peer-count side.

Any mitigation that turns off DeltaQ machinery "on inbound because no Leios logic reads it" is therefore wrong — it silences the exact tx-submission signal Frame 2's pre-diffusion diagnostic depends on (see The failure-mode diagnostic table). Direction-based gating is not a safe cost-reduction strategy.

Mitigation strategies

Ordered by leverage. Choose a combination based on which frame you're supporting and how much per-peer fidelity the consumer actually needs.

1. Statistical sampling across inbound peers.

Frame 2's pre-diffusion diagnostic asks a mesh-aggregate question — "is gossip healthy across my peers?" — not a per-peer one. Instrument a random 30–50 of 500 inbound peers rather than all 500. Statistically indistinguishable for aggregate purposes at ~10x lower cost. Rotate the sample every few minutes to avoid systematically favouring a subset.

Does not help if the consumer needs per-peer signals, but for the pre-diffusion diagnostic specifically it's a large, safe reduction.

2. Protocol-scoped enablement.

Only run cookie/burst machinery for mini-protocols whose signals are consumed:

  • Tx-submission on inbound → for pre-diffusion monitoring.
  • Block-fetch on outbound → for peer selection.
  • Skip chain-sync (no consumer for its RTT signal here) and keep-alive (already provides a lightweight RTT signal, doesn't need cookies).

Roughly halves per-connection state and per-SDU cost by trimming the per-MiniProtocolNum map to just the protocols in use.

3. Probe sampling per SDU.

Cookie 1-in-N SDUs rather than every SDU. Reduces PSQ churn linearly. RTT is stationary enough that 1-in-4 or 1-in-10 sampling still gives hundreds of samples per minute per peer — well above KeepAlive's 6-per-minute baseline.

4. Aggregate-only mode.

For consumers that only ever ask "median across my peers", collapse the per-peer RTT windows into a single shared window per protocol. Trade per-peer visibility for cost. Not compatible with peer-selection use cases (which need per-peer numbers) but fits Frame 2 pre-diffusion monitoring cleanly.

5. Compile-time flag.

Gate the whole Network.Mux.RTT module behind a deltaq_rtt cabal flag, mirroring the existing tracetcpinfo pattern. Off by default. Ultimate opt-out for operators unwilling to accept any DeltaQ cost.

Downstream consequence: a node with the flag off cannot participate in any Frame-2 behaviour. This bifurcates the network into "Frame-2-capable" and "vanilla" nodes — a real deployment concern requiring governance consideration, not just an ops decision.

6. Consumer-side aggregation.

Move StatsA aggregation off the mux path; ship raw TraceRecvDeltaQObservation and TraceRecvBurstSDU events to cardano-tracer and aggregate at the consumer. Trades in-mux CPU for trace-forwarding bandwidth. Practical for testnet measurement nodes where the consumer is close by; impractical for production nodes where raw per-SDU trace volume would swamp the forwarder.

What this means for the two Leios frames

Frame 1 (validation / parameter driving) runs on ~30–50 instrumented measurement nodes with a bounded connection count. Full machinery is affordable at that scale — no mitigation needed for the campaign itself.

Frame 2 (adaptive Leios in production) requires the machinery to run on public relays with hundreds of inbound connections. A defensible production configuration:

  1. Statistical sampling across inbound peers (~30–50 of ~500), with rotation.
  2. Protocol-scoped enablement (tx-submission on inbound, block-fetch on outbound).
  3. Probe sampling at 1-in-4 or 1-in-10.
  4. Compile-time flag available as the final opt-out for operators who refuse the cost.

The compile-time flag creates a network bifurcation. Nodes with the flag off can't participate in Frame-2 behaviours (voter self-throttle, adaptive S_EB_tx, runtime DeltaQ report's Rec 3 enforcement). Whether Frame-2 behaviours are advisory (nodes MAY adapt) or mandatory (nodes MUST adapt) is a CIP-0164 governance question the branch doesn't answer.

The mitigation-vs-safety interaction

Frame 2 changes the shape of the network's protocol behaviour in a way Frame 1 does not. In Frame 1 the branch only measures — nodes either produce more or less data, but the protocol is unchanged. In Frame 2, nodes voluntarily deviate (throttle, opt out) based on local observation. That deviation is only sound if the deviation criteria are so rare that variation doesn't move P_cert, or if every node's deviation happens under sufficiently correlated conditions.

The cost mitigations above interact with this. Statistical sampling introduces variance in what each node observes, which introduces variance in when each node throttles. If observation is uncorrelated across nodes, the network gets graceful degradation. If observation is correlated (all nodes throttle at the same moment because they observe the same shock), the mesh loses quorum en masse and P_cert collapses. This is the cross-node covariance question from the blended-diffusion section, applied at the level of adaptive decisions rather than pre-diffusion state.

So cost mitigation isn't just an engineering optimisation — it interacts with the operational safety of the Frame-2 lever. That interaction is worth measuring on the testnet before committing to any particular mitigation combination in production.

Adaptive Leios via the PeerRTT reader view

What the reader view actually enables

Static vs. adaptive parameter setting — the category shift

The Leios report and CIP-0164 both operate in what one might call the static-analytical mode: pick network conditions (RTT distribution, congestion model, loss rate), solve for parameters (L_vote, L_diff, S_EB_tx) such that P_cert ≥ threshold, freeze those parameters into the protocol. Once frozen, every node applies them identically, blind to whether the actual mesh conditions at that moment resemble the assumed ones. A voter with degraded upstream RTT still votes on the same deadline as a voter on a healthy link; a block producer still packs closures to the same S_EB_tx regardless of what its actual peers can absorb.

The PeerRTT reader view breaks this because a live Leios voter (or block producer, or non-voter fetcher) can read, at runtime, what the actual RTT quantile is right now to each of its peers. That's a category shift from offline-analytical to online closed-loop. The mini-protocol suddenly has a signal that lets it be a proper control system rather than an open-loop feed-forward one.

Concrete adaptive behaviours it enables

Ordered roughly by conservatism:

  1. Self-diagnosis / operator alerting. The mildest use. Compare the running node's observed PeerRTT.quantile 0.95 against the values CIP-0164 assumed. If measured p95 exceeds assumed p95 by some margin, log a warning. Doesn't change protocol behavior; just gives operators a signal that their node is operating outside the CIP-0164 design envelope.

  2. Adaptive peer preference. BlockFetch already selects peers by PeerGSV, but its update path is via KeepAlive's ~10 s cadence — coarse. The window-based signal from this branch is per-connection, updated on every response, and covers all miniprotocols. Peer-selection could react to degradation seconds faster, and could demote peers whose measured RTT has drifted from the initial handshake expectation.

  3. Adaptive S_EB_tx sizing at production time. A block producer preparing an EB knows, from its peer RTT window, roughly how much bandwidth it has budgeted for closure diffusion. If p95 RTT to typical peers has doubled since CIP-0164 assumed values, the producer could cap the closure size below S_EB_tx_max — voluntarily accepting a smaller EB to keep certification probability up. Global CIP-0164 sets a ceiling; the node sets the actual size per-round from observations. This is also the mesh's stabilizing feedback against sustained-overload livelock — see the Cascade to livelock section below.

  4. Adaptive Freshest-First Delivery (FFD). FFD is the "when I'm behind on diffusion, prioritise the newest EB and drop the older" mechanism the report defers modelling (§7 limitation 7). The trigger for FFD is fundamentally "am I going to make my deadline?" — which requires knowing (a) time remaining, (b) work remaining, (c) achievable throughput. Item (c) is exactly PeerRTT + estBurstS. Static FFD thresholds are guesses; adaptive FFD becomes possible with a reader view.

  5. Voter self-throttling. The most aggressive use. A voter observing PeerRTT.quantile 0.95 far above the CIP-0164 assumption could conclude "I have no realistic chance of completing the full pipeline in 7 s" and refuse to vote for that round — better than voting and later failing to diffuse the certRB. This trades quorum size for quorum quality: fewer votes, but the ones cast are more likely to actually diffuse.

The interesting property that runs through 3–5: they let a node voluntarily degrade under stress in ways the static analysis can't. The report's P_cert collapses to ≈ 0 under catastrophic pre-diffusion failure because every voter tries and every voter fails. An adaptive scheme where stressed voters bow out could keep the quorum threshold met among the healthier subset — turning a hard cliff into a graceful degradation.

Why this doesn't come for free

Three problems, in decreasing order of severity:

  1. Adversarial gaming. A peer that deliberately delays echoing our cookies inflates our locally-observed RTT to that peer. If we react by demoting the peer, that's fine (self-correcting). But if we react by throttling our own participation (self-throttling, item 5), a small colluding set of peers can push a targeted voter out of the committee. Mitigation: aggregate across peers (a single bad peer only pollutes one bucket in the RTT window), and require the degradation to be broad — measured against the median of our peers, not the worst.

  2. Feedback loops in the mesh. If some nodes self-throttle and others don't, the ones still voting have to carry more of the diffusion load, potentially degrading the observed RTT for them, which could cascade. This is a classic control-theory pitfall. Any adaptive behavior needs damping — a hysteresis band, or a stochastic component in the throttle decision — to avoid synchronized oscillation across the mesh.

  3. Governance / spec-compliance. CIP-0164 is a consensus-critical spec; adaptive voter behaviour that varies decisions based on local observation is a soft deviation from "every voter runs the same protocol." Whether an adaptive voter's cast/not-cast decision is spec-compliant, or whether the CIP needs to explicitly allow (or mandate) it, is a governance question — not a technical one. The technical prerequisite (the reader view) is there; the political prerequisite isn't.

Why it's specifically interesting for Leios

Praos already has adaptive elements (peer selection, BlockFetch churn, ChainSync candidate ordering) that consume PeerGSV. So the pattern — miniprotocol reads network telemetry, adjusts behavior — is not new to Cardano. What's new is:

  • Leios's deadlines are tighter. Praos has a slot budget of 20 s and typically delivers within 1–2 s; a factor of 10 headroom absorbs a lot of RTT variance without behavioral change. Leios's 7 s voter deadline against a 4.4 s modelled long-haul fetch has a factor of ~1.6 headroom. Variance that Praos absorbed silently starts to matter.

  • Certification is atomic. A slow block in Praos still gets adopted eventually; a slow EB simply doesn't certify. There's no "eventually" — either you make the 7 s deadline or you don't. That's exactly the regime where local decision-making beats global static settings.

  • The report's own recommendation 3 depends on it. "Ensure effective tx-submission pre-diffusion" (report.md:1584-1588) is currently unsatisfiable at protocol design time because pre-diffusion is a runtime property. But it's satisfiable at runtime: a node observing degraded pre-diffusion (via per-mini-protocol DeltaQ traces on tx-submission) can react — refuse to vote, reduce closure size, alert. The PeerRTT reader view (and its per-protocol siblings) is the API that turns Recommendation 3 from unsatisfiable to enforceable.

The short answer

The reader view means Leios doesn't have to bet its safety on the network conditions assumed at CIP-0164 design time being the ones present at runtime. It can measure, and adjust. That's a lever the report's static-analytical framing doesn't consider, and probably the single largest place where the branch's work changes what Leios can do, as distinct from what the report can verify.

Fully blended diffusion

The reader-view section covered adaptive protocol behaviour in the good case, where the mesh is broadly healthy and a node is fine-tuning around CIP-0164's assumed baseline. This part covers the degraded case: when 1-hop pre-diffusion has failed and the full EB closure has to traverse the multi-hop mesh, with the report's α · P_{cert,1-hop} mixture collapsing towards the blended-diffusion tail. Two subsections — one on diagnosing which failure mode you're in, one on investigating the dynamics of the mode itself.

Diagnosing pre-diffusion: transport vs. outcome

The report's Limitation 1 and Recommendation 3 both hinge on pre-diffusion — the assumption that transactions have already reached local mempools via tx-submission gossip before the EB referencing them is produced. Testing whether that holds, and diagnosing why when it doesn't, needs two signals from different layers. Neither is sufficient on its own; conflating them (as the report's α mixture parameter implicitly does at report.md:1584-1588) loses information a production node needs.

The two signals

Transport-layer (mux-layer, from the branch's traces). Per-mini-protocol DeltaQ traces bucketed to tx-submission give per-peer RTT for RequestTxIds → ReplyTxIds and RequestTxs → ReplyTxs exchanges, throughput of ReplyTxs bodies (bytes/sec of tx traffic actually flowing), and channel liveness (is the peer responding at all?). Interpretable as "how healthy is the tx-gossip pipeline right now, per peer?"

Application-layer (from EB processing, not mux). At each EB arrival, cross-reference the EB's referenced tx-hash list against the local mempool. The fraction not present is the observed π₁ for that EB, at this node, at this moment. Interpretable as "how out of sync was my mempool at the moment I needed it?"

The report's mempool-measurements dataset already collects the outcome signal at coarse granularity (per-node, per-block). The mux traces contribute the transport signal, per-connection, at higher temporal resolution. Both are needed.

The failure-mode diagnostic table

Composing the two gives a diagnostic that distinguishes qualitatively different failure modes:

Transport health Observed π₁ Interpretation Response
Normal Normal Healthy pre-diffusion. —
Degraded Elevated Slow gossip → mempool lagging. The characteristic transient-stress signature (peer churn, bufferbloat, brief network hiccups). Prefer healthier peers via peer-selection churn; wait it out. Escalate if severity persists or grows (see below).
Normal Elevated Pipeline fine yet mempool out of sync. Points at adversarial withholding, a targeted partition affecting only tx-submission for a subset of peers, or a Cardano-specific gossip pathology. Alert operators; refuse to vote in the affected round; force reconnect to fresh peers. Distinct from network stress and warrants a stronger response.
Degraded Normal Slow gossip but overall state healthy — a caching/burst effect, or a lagging monitor whose window hasn't caught up. No action; sanity-check the transport metric.

The Degraded × Elevated row deserves a severity distinction in practice. A mild version — transport RTT up 2×, π₁ from 0.06 to 0.15 — is the transient signature this row identifies. A severe version — transport RTT up 10×, π₁ from 0.06 to 0.4 — is a compound failure warranting the same response as Normal × Elevated (throttle production, refuse to vote). Baseline calibration on the testnet is what tells you where "mild" ends and "severe" begins for each region-pair.

The value of the split is that "pre-diffusion is failing" is currently a single conflated signal in the report. A production node armed with the transport-vs-outcome split can react to which kind of failure is in play, and reserve the strongest responses (refusing to vote, alerting operators) for cases that actually warrant them.

Baseline is a prerequisite

The table's "Normal" and "Elevated" categorisations are only useful with a per-region-pair baseline distribution — mean and tail of tx-submission RTT and throughput, mean and tail of observed π₁, all measured under representative load. A week of the testnet campaign's baseline collection (see What a good testnet campaign looks like above) establishes those baselines. Without them, the categorisation is guesswork.

Forced-blended experiments in the next section intentionally drive the mesh into the Normal × Elevated quadrant on the suppressed nodes — their tx-submission is fine with peers that also don't suppress, but their local π₁ shoots up. Confirmatory: the diagnostic table should reproduce the induced failure mode. If it doesn't, either the induction is not doing what you think or the diagnostic is not measuring what you think — both worth debugging.

Investigating fully-blended diffusion when 1-hop fails

Collected and analyzed DeltaQ data fits well as inputs to the report's blended-diffusion model, indirectly but usefully covers the composed end-to-end multi-hop dynamics, and — most valuably — can reveal a family of dynamics the report itself doesn't model.

What "fully blended diffusion" actually requires empirically

The report constructs its blended-diffusion CDF (report.md:359-436) by convolving:

  1. A per-hop transfer-time CDF, parameterised by the throughput model (Mathis or CUBIC) at assumed p and RTT bin (short/medium/long).
  2. A path-length distribution (1..5 hops with the report's regular-random-graph probabilities).
  3. An RTT-bin mixture (short/medium/long equally likely per hop).

Every piece of that has to hold up empirically for the report's conclusion (14 s delivery only 14% under Mathis, 99% under CUBIC) to reflect reality.

What DeltaQ traces give you directly

The single-hop CDF — item 1 above — is exactly what per-connection estBurstS and PeerRTT produce. On a real testnet:

  • Every burst response (blockfetch chunk, EB body, missing-closure fetch) generates an estBurstS sample: throughput of that hop under the actual congestion regime, actual p, actual buffering. No Mathis/CUBIC assumption in sight.
  • Every matched cookie produces an RTT sample; a PeerRTT quantile per peer gives you empirical per-hop CDFs binned by peer-distance category.

Plugging those CDFs into the report's convolution instead of the analytical Mathis/CUBIC ones gives you a directly-empirical estimate of the blended-diffusion CDF at each S_EB_tx. That's the cleanest validation path: keep the report's convolution engine, replace its analytical inputs with measurements, compare outputs.

This alone would settle whether the 14% (Mathis) or 99% (CUBIC) figure at 12 MB is closer to reality on a given mesh.

What only needs a topology snapshot (cheaper than correlation)

One blended-diffusion input is answerable from the peer graph alone — no DeltaQ traces, no per-EB tagging, no consumer plumbing. The main one is structural path-length distribution, the assumption the report leans on hardest for its blended-diffusion CDF (report.md:359-436).

Operational shape:

  • Query each testnet node's peer-selection state (already introspectable via existing node interfaces).
  • Dump the graph — edges are directed peer connections; vertices are nodes.
  • Compute the shortest-graph-path distribution, degree distribution, clustering coefficient, and hub concentration.

What it answers. Does the mesh's structure resemble the report's regular-random-graph assumption (degree ~10, no significant clustering, path-length ~3.6)?

What it does not answer. What paths EBs actually take. Peer-selection preferences and BlockFetch decisions can route diffusion onto longer or shorter paths than the shortest-graph-paths. That gap is filled by reconstruction (next subsection).

Ordering the two. Topology snapshot first, reconstruction second:

  1. Topology snapshot is a one-day exercise. Immediate feedback on whether the mesh's structure resembles what the report assumes. If the answer is "no", that alone is a finding worth reporting before running the more expensive reconstruction. Repeat weekly on a live testnet to catch topology drift.
  2. Reconstruction (next subsection) is the higher-fidelity but more expensive measurement, and its number is what the report's convolution actually depends on.

The comparison between the two produces one of four outcomes, each telling a different story about what should happen next:

Topology says Reconstruction says Interpretation Follow-up
Matches assumed ~3.6 Matches topology Mesh is what the report assumed; peer-selection isn't biasing diffusion. Report's convolution well-founded; confidence in 12 MB feasibility increases.
Matches assumed ~3.6 Longer than topology (e.g. ~4.8) Structure is fine but peer-selection pushes diffusion onto longer paths. Investigate peer-selection behaviour; unexplained bias in the diffusion routing.
Longer than assumed (e.g. ~5.2) Matches topology Mesh structure is worse than assumed; diffusion follows what the structure allows. Recalculate the report's convolution with the empirical path-length.
Longer than assumed Shorter than topology Peer-selection is routing around long paths (via hubs, for instance). Model has a hidden resilience mechanism worth documenting.

Doing only one of the two measurements leaves ambiguity that the other resolves.

What requires cross-node correlation

The convolution assumes hops are independent and identically distributed. The mesh may not honour that. To check, you need to observe end-to-end delivery, not just single-hop:

  • Tag every EB (or synthetic bulk payload) with a stable ID, log (node, ts, EB_id, arrival_or_forward) events across the whole testnet.
  • Also tag each BlockFetch / LeiosFetch fetch session at each node with a local session ID that references the EB hash it's pulling. The EB-arrival log above is application-layer and answers "when did EB X arrive at node Y"; a per-fetch session tag on the local traces answers "which mux-layer signal belongs to which fetch". Under the interface-saturation scenario in Dynamics the report doesn't model (item 1), this makes the cross-node correlation direct: "during session $S$ for EB $X$ on connection B↔C, B's other Wantons saw oldest-byte-age rise by $\Delta$, and peers D/E/F saw estBurstS drop by $\varepsilon$" — instead of timestamp-guessing which spike belongs to which fetch. No wire-format change needed: the EB hash is already on the wire (BlockFetch / LeiosFetch requests reference EBs by hash), so the session ID can be derived deterministically from (peer, EB_hash, fetch_start_time) and only needs to appear in the local BlockFetch / LeiosFetch client trace stream to key mux-layer signals against.
  • Reconstruct actual paths by matching each node's arrival time against its forwarding events, plus the known peer graph. You now have the actual multi-hop trajectory of each EB across the mesh.
  • Compare the observed end-to-end arrival CDF against the DeltaQ- convolved prediction. If they diverge, the independence assumption is wrong — head-of-line effects at intermediate hops, or graph-topology skew, are the likely culprits.

This isn't in the mux traces themselves — it's a higher-level analysis layered on top. But the mux traces plus a modest per-EB-arrival log gives you enough to do it. cardano-tracer already emits block-arrival events; the same pattern for EBs would suffice.

Measuring $F_{\text{full}\mid C}$ directly (Yves Hauser's §5.6 target)

The same per-EB arrival log that enables path-length reconstruction also lets us directly measure Yves Hauser's §5.6 conditional probability: given certification succeeded at $t_v = 7\text{ s}$, did all $N$ honest nodes receive the EB body by $t = 14\text{ s}$? See leios-conditional-diffusion.md for the full analysis; the operational recipe is:

  • Log (node, EB_id, arrival_ts) at every measurement node (same log as the cross-node correlation section above).
  • Log certification outcome per EB (from cardano-tracer's block-diffusion / EB-certification traces).
  • For each EB $j$ with $C_j$ (certification succeeded), check whether $\max_i T_j^{(i)} \le 14\text{ s}$.
  • Empirical estimator: $\hat{F}_{\text{full}\mid C}(14) = \frac{\left|{j : C_j \text{ AND all arrived by 14 s}}\right|}{\left|{j : C_j}\right|}$.

Yves's formula predicts $\approx 1.000$ at the report's default parameters. The testnet gives us an empirical rate. Detection tail power: ~30 k EBs per week distinguishes 1.000 from 0.999 comfortably, 0.999 from 0.99 borderline, 0.99 from 0.9 easily.

As a by-product, the same data gives:

  • The identity of the slowest node per EB ($\arg\max_i T_j^{(i)}$) and the temporal regime of that identity across rounds (persistent / rotating / bimodal). Per-node operational intervention pays off only under the persistent regime; the rotating and bimodal (churn-induced) regimes each call for different levers. See leios-conditional-diffusion.md's "Slowest-node identity and its temporal structure".
  • Empirical vs modelled $G(t)$ — divergence tells us whether $G$ is over- or under-estimated at each body size.
  • The correlation matrix between per-node arrival times — directly addresses Yves's caveat 1 (i.i.d. assumption) and gap 2 of leios-report-gaps.md.

Measuring churn-induced peer correlation and orphan-set geography

The appendix in leios-fetch-scheme.md identifies a tendency for peer-selection churn to concentrate the mesh's big-ledger peer sets around block-production regions, producing bimodal per-round coverage (high on dense-origin rounds, low on sparse-origin rounds). The actual strength of this tendency on real networks is an empirical question. The same per-EB arrival log used for $F_{\text{full}\mid C}$ measurement (above), extended with peer-metadata and block-origin metadata, supports the validation.

1. Per-round orphan-set identification. For each certified EB, identify the set of nodes that did not receive delivery from a big-ledger seed (their winning arrival was via a non-big-ledger path, or arrived after a threshold indicating multi-hop). Peer-metadata (big-ledger flag per connection) distinguishes the winning path.

2. Block-origin classification per round. For each certified EB, identify the block producer's geographic / ASN region. Classify the round as dense-origin or sparse-origin based on the producer's location (or, more robustly, based on stake-weighted region density in a rolling window). This is the key axis of the analysis.

3. Per-origin coverage rates ($c_D$, $c_S$, empirical $k$). Compute mesh-wide seed-hop rate separately for dense-origin and sparse-origin rounds:

  • $c_D$ = fraction of dense-origin rounds where most of the mesh had seed-hop delivery.
  • $c_S$ = same for sparse-origin rounds.
  • Empirical $k = c_S / c_D$.

If $c_D \approx 85%$ and $c_S \approx 40%$, then $k \approx 0.47$ — strong-ish concentration. If $c_S \approx 75%$, then $k \approx 0.88$ — weak concentration, the tendency isn't dominant.

4. Per-round bimodality analysis. Instead of asking "which nodes are persistently orphaned", ask "which rounds have widespread orphaning". Histogram the mesh-wide orphan fraction across all rounds. Under the churn tendency, expect a bimodal distribution (peak near zero for dense-origin rounds, peak near 0.75–0.8 for sparse-origin rounds). Under weak tendency, expect a unimodal distribution centered on the mesh average.

5. Cross-node covariance within rounds. On sparse-origin rounds, compute the covariance between arrival times across nodes. High within-round covariance is the signature of a shared bottleneck (the multi-hop diffusion route from a sparse origin). This directly measures Yves's caveat 1 applied to the seed-node lottery.

6. Peer-graph diversity metrics (secondary). For each node, compute ASN / country / geographic diversity of the 5 big-ledger peers, peer tenure, and average RTT. Correlate diversity with per-node orphan-frequency. Under the tendency, diversity should correlate positively with fewer sparse-origin orphanings; under weak tendency, no strong correlation.

7. Effect on $F_{\text{full}\mid C}(14)$ split by round type. Compute empirical $F_{\text{full}\mid C}(14)$ separately for dense-origin and sparse-origin rounds. If sparse-origin $F_{\text{full}\mid C}$ is significantly below dense-origin, the tendency is materially degrading protocol safety on those rounds. Operational trigger for deploying Improvement #8 (diversity-aware churn) or accepting the reduced robustness on sparse-origin rounds.

Measurement pre-requisites

  • Peer-metadata service — MaxMind or equivalent for ASN / country lookups from IP addresses. Roughly a few MB static database, daily refresh. Cheap.
  • Peer-graph tracking — per-node record of which peers were big-ledger vs NBL at each round. Small addition to the per-EB arrival log.
  • Block-producer identity per round — required to classify rounds as dense-origin or sparse-origin. Available from the block header / chain-sync metadata already emitted by cardano-tracer; needs to be joined into the per-round dataset.
  • Regional partitioning of stake — a rough "dense vs sparse" region cutoff for classifying block-origins. Could be geographic (continents), by-ASN, or by empirical stake density. Multiple partitionings for robustness against a bad partition choice.

Comparison across configurations

Under the four-configuration experiment (item 6 of "What a good testnet campaign looks like"):

  • Memo baseline: establish empirical $c_D$, $c_S$, $k$, and the per-round bimodality profile. If $k \approx 1$ (no strong concentration), the tendency isn't dominant on this mesh and the other configurations may not add much. If $k \ll 1$ (strong concentration), we have a real problem to fix.
  • +Improvement #1 (hybrid classification): minimal effect on churn dynamics — classification refines within the existing peer set but doesn't change churn selection. Concentration should be similar to baseline.
  • +Improvements #1–#8 (full adaptive): Improvement #8's diversity-aware churn should measurably raise $c_S$ (sparse-origin coverage) and hence $k$. Regional variance in $F_{\text{full}\mid C}$ across round-types should shrink.
  • Erasure-coded variant: orphaning behaviour differs entirely (no single seed peer needed). Different question — not directly comparable but useful as an alternative.

The dominant question the measurement answers: how much does diversity-aware churn actually raise coverage on sparse-origin rounds? If the answer is "meaningful gain" ($k$ climbs from 0.5 to 0.8), the mitigation is worth deploying; if "marginal" ($k$ climbs only 0.5 → 0.55), the complexity may not be justified.

What requires inducing the scenario

Blended diffusion only matters when 1-hop pre-diffusion has failed. To exercise it on a testnet you have to force pre-diffusion to fail:

  • Suppress tx-submission on a controlled subset of nodes (or the whole mesh) for a window. Every EB in that window has to be blended-diffused.
  • Introduce artificial mempool divergence by having a fraction of nodes accept transactions from a private endpoint that others don't see. π₁ rises artificially; blended-diffusion becomes the norm.
  • Withhold: an adversarial-simulation variant where a bloc of nodes deliberately don't forward tx-submission but do forward blocks. Tests the report's §7 limitation 5 (adversarial withholding).

Under any of these, DeltaQ traces then show you the actual blended-diffusion dynamics under load. The report predicts what should happen; the trace shows what does.

Dynamics the report doesn't model, but DeltaQ traces expose

This is where DeltaQ trace collection is most valuable, because it can challenge the report's framing rather than just confirming its numbers.

  1. Inter-connection interface contention at forwarding nodes. When node B is forwarding a 12 MB closure from A to C, the fat LeiosFetch flow to C consumes a large fraction of B's outbound interface bandwidth. B's concurrent thin flows on other connections — tx-submission to D, chain-sync to E, KeepAlive echoes to F — share the interface with the fat flow. Note this is not intra-connection head-of-line blocking (the mux is fair at the SDU level, so tx-submission on the B↔C connection interleaves with LeiosFetch SDUs, not queues behind them). It's inter-connection competition for B's shared network interface, which degrades the thin flows only if the interface is genuinely saturated. Detecting this can go via cross-node correlation or local self-monitoring at B, using different signals:

    • Cross-node. estBurstS is receiver-side — each of B's peers logs its own estBurstS for traffic from B, and a simultaneous drop across multiple peers of B during B's bulk-transfer window is the signature.
    • Local at B (self-monitoring). Under NIC saturation, kernel TCP throttles B's socket-to-D (the thin flow), the socket's send buffer fills, and the mux write from the tx-submission Wanton blocks — so bytes accumulate in the Wanton and oldest-byte-age on B's per-mini-protocol egress queue grows. A rising oldest-byte-age on B's thin-flow Wantons during B's fat-flow window is the local signature of interface saturation — per-mini-protocol, per-connection granularity, exactly what we need. The dual signal is socket write-block latency: time the mux egress thread spent blocked on a socket write with the kernel send buffer full — the mux-visible echo of the kernel-side congestion. Both traces are proposed in track.md's "Measurements blocked on missing traces" (items 2 and 6) but not yet emitted; adding them would give B direct self-monitoring for this scenario, covering both sides of the stall (queue backing up, mux blocking on the socket). Complementary signals available today: aggregate outbound bandwidth from OS-level interface counters, and TCPInfo (tcpi_snd_cwnd, tcpi_retrans, tcpi_lost) which shows the kernel-side congestion state driving the mux stall.

    The report models per-hop transfer time as if each connection has independent access to the wire (see leios-report-gaps.md gap 5 — per-node interface contention across concurrent connections is not modelled).

  2. Round-to-round feedback. A round with heavy blended diffusion consumes mesh bandwidth that would otherwise carry tx-submission, which raises π₁ for the next round, which raises the blended-diffusion load again, which... The report treats rounds as independent Bernoulli trials. Time-series DeltaQ traces across many consecutive rounds show whether the mesh has this positive-feedback autocorrelation. If it does, the tail probability of consecutive-round failure is worse than the report's Bernoulli model implies.

  3. α as a time-varying observable rather than a scalar. The report defers α (probability that pre-diffusion is operating normally) to a single number in the α · P_cert,1-hop mixture, and treats it as a system-wide constant. But at each moment, per node, the observed fraction of arriving EBs whose π₁ stays below the 1-hop threshold is a direct sample of the α distribution — computable from EB arrival processing joined against local mempool state. That converts a hidden static parameter into a time series with a distribution, a tail, and cross-node covariance — the last of which is the important one: whether pre-diffusion failures hit the mesh in lockstep or independently is the difference between a global P_cert collapse and quorum still meeting from healthier nodes, and the report's single-scalar α silently assumes the lockstep case.

  4. Recovery dynamics. After a pre-diffusion failure, how does the mesh recover? Do voters flag the miss and re-fetch aggressively? Does the tx-submission backlog burst-catch-up? DeltaQ traces around the failure event show the recovery signature — most importantly, whether recovery is monotone or oscillatory. Monotone recovery is safe; oscillation is a warning about latent instability.

  5. Storm dynamics under concurrent EBs. Nothing forces exactly one EB per round to be certified concurrently — there could be multiple candidates propagating simultaneously. DeltaQ traces during multi-EB windows probe whether the mesh gracefully shares bandwidth or whether one EB starves the others.

  6. Path-length skew. The report assumes a regular random graph. Real testnets have topology bias (regional clustering, hub nodes, degree variance). Reconstructed EB paths from cross-node correlation reveal the actual path-length distribution and whether it matches the report's assumed distribution. If typical paths are actually 4.5 hops rather than 3.6, the whole convolution shifts.

Cascade to livelock: bounding sustained-overload failure

Item 2 above (round-to-round feedback) names the mechanism; this subsection names the pathological outcome and enumerates the measurements that bound it.

The failure mode. Sustained mempool arrival at or above the level at which max-size EBs are produced imposes network cost (EB body diffusion, closure fetch, vote diffusion, certRB diffusion) and CPU cost (apply/reapply for validation) each round. If certification fails at that operating point, the work is not wasted — TxCache retains the TX bodies for reuse — but the transactions themselves stay in the mempool, the next producer sees a similar-or-larger backlog, and produces another max-size EB. If the reason certification failed is a saturation in the components TxCache does not amortize (network capacity for EB body / votes / certRB, per-node validation CPU, slow-node tail dominance of $F_{\text{full}\mid C}(14)^N$), the next round's certification fails for the same reason. The mesh is doing work each round but not making protocol progress — a livelock.

TxCache is necessary but not sufficient. TxCache amortizes the TX-body diffusion component of the closure and (if it caches validation state deltas) some of the apply/reapply cost. Without it, retries pay full closure and full validation cost and the livelock enters much faster. But TxCache does not add capacity to the pieces it doesn't cover:

  • The EB body itself is a distinct network artifact and diffuses per round regardless of what's in TxCache.
  • Vote and certRB diffusion are per-EB, TxCache-independent.
  • Interface-level network saturation (per leios-report-gaps.md gap 5) is a capacity problem TxCache doesn't touch.
  • Persistent slow nodes dominating $F_{\text{full}\mid C}(14)^N$ don't get faster because their TX-fetch was cached.
  • Under bursty multi-producer EBs with disjoint TX sets, cache locality erodes exactly when it's most needed.

Under sustained saturation on any of these axes, TxCache lowers the marginal cost of a failed round but does not raise the certification success rate above the underlying capacity ceiling.

The stabilizing lever is adaptive body sizing at production time, not TxCache. A producer observing degraded mesh health (previous-round certification outcome, local PeerRTT / estBurstS degradation, mempool-vs-pre-diffusion gap) voluntarily caps $S_{EB\text{-tx}}$ below the CIP-0164 ceiling — cutting the per-round load until certification recovers. See leios-conditional-diffusion.md's lever 2 and the reader-view section above. Voter self-throttle (runtime Rec 3 enforcement, lever 3) is complementary but weaker — it prevents committing to failing rounds but doesn't reduce the load producing them.

TxCache's role in the safe regime is to make the transient retry period survivable while the adaptive lever converges to a sustainable body size. Under a well-tuned adaptive-sizing loop, TxCache is the amortization that keeps retry cost bounded during convergence. Under a mis-tuned or absent adaptive lever, TxCache alone does not prevent the runaway.

Measurement recipes

Ordered from cheapest (existing traces) to most involved (induced- failure experiments).

Detecting the feedback loop. Available today or with minimal instrumentation.

  • Per-round certification outcome autocorrelation. From cardano-tracer's EB-certification traces, compute $\mathrm{Corr}(\mathrm{fail}(R), \mathrm{fail}(R+k))$ for $k = 1, 2, \ldots$. Positive autocorrelation is the cascade signature; the decay rate quantifies memory depth. The independent-Bernoulli null hypothesis gives $\mathrm{Corr} = 0$.
  • EB body size vs prior-round outcome. Cross-tabulate round $R+1$'s body size against $\mathrm{fail}(R)$. If body size systematically grows after a failure, that's the direct feedback signature. Available today from block-production traces.
  • Mempool depth trajectory across consecutive-failure runs. Does depth grow monotonically during a failure sequence, or does it stabilize? Unbounded growth signals the runaway. From the mempool-measurements dataset.

Quantifying TxCache's role. Splits the retry-cost savings attributable to TxCache from other sources.

  • Per-node TxCache hit rate, split by "TX belongs to a failed EB from round $R-k$" vs "TX arrived via ordinary tx-submission". Reveals what fraction of retry savings is TxCache's contribution.
  • Closure size on retry. For consecutive EBs sharing overlapping TX sets, measure the actual closure-fetch bytes pulled at each node. Reduction vs full closure is TxCache's realised savings — per node, per round, per overlap fraction.
  • CPU cost of apply/reapply on retry, split by cache-hit status. If TxCache caches validation state, retry cost should drop sharply; if only TX bodies, revalidation cost is largely unchanged. Uses existing post-cip/apply-reapply instrumentation.

Finding the collapse threshold and recovery signature. Controlled experiments that induce the failure mode.

  • Sustained-load-to-collapse sweep. Increase tx-submission arrival rate until certification failure rate crosses a threshold (e.g., >5% consecutive failures over a rolling window). Drop the arrival rate back to baseline and measure whether certification recovers. If certification stays stuck after load drops → livelock confirmed. Report the smallest load level above which recovery does not occur (the "livelock onset" boundary).
  • Induced-failure recovery experiment. Force a round to fail — withhold votes at a controlled slice of the committee, or inject a controlled network shock via tc netem — then observe recovery: how many rounds until certification returns to baseline, and whether the trajectory is monotone or oscillatory. Signatures per item 4 above.
  • Time-to-recovery distribution. Across many induced failures at different load levels, measure $\Pr(\text{recovery within } k \text{ rounds})$ as a function of load. A regime where recovery time diverges beyond the tx-arrival timescale is where livelock lives.

Tuning the stabilizing lever. Once producers use adaptive body sizing, calibrate its response.

  • Adaptive body sizing calibration. At the collapse threshold, how much body-size reduction restores certification? Report the operating curve $S_{EB\text{-tx}}(\mathrm{load})$ that keeps the certification success rate above a target.
  • Adaptive-sizing loop stability. When producers throttle, next-round body size shrinks, certification recovers, mempool grows, next-round producer sees lower degradation, body size grows again. Whether this loop converges to a fixed point or oscillates is the open question flagged in leios-conditional-diffusion.md's follow-ups. Measure the loop-gain from perturbation experiments.
  • Voter self-throttle activation rate. How often does the self-throttle fire, and does the fire rate correlate with load? A throttle firing so often that certification never happens is a different failure mode (throttle-induced stall) but it's how the mesh avoids the livelock. Measurable from voter-participation traces once the runtime lever is enabled.

Cross-referenced signals from other measurements.

  • Per-EB arrival log with certification outcome — the same log powering $\hat{F}_{\text{full}\mid C}(14)$ (per leios-priority-measurement.md).
  • estBurstS and PeerRTT degradation cross-referenced against the failure-autocorrelation series — links the transport-layer signal to the round-outcome signal.
  • Interface-saturation self-monitoring (oldest-byte-age and socket-write-block latency, per track.md items 2 and 6) during heavy-round windows — signals whether the network layer is the binding constraint.

Cross-references

The practical shape of a "blended-diffusion investigation" campaign

If this is your goal, the testnet setup differs slightly from the plain validation campaign above:

  • Explicit forcing knob: a scripted way to enable/disable tx-submission per node. Cardano-node already supports peer-filtering hooks that could do this.
  • Per-EB arrival log: augment cardano-tracer output to include (node, EB_id, arrival_ts, forward_target_ts) records. Not currently emitted per-EB but the trace event exists at block-diffusion level.
  • Round-cadenced captures: raw DeltaQ traces on measurement nodes during a specific window around a forced-blending event, not just steady-state sampling. Higher temporal resolution during the interesting events, aggregated the rest of the time.
  • Multi-round runs: at least 100 rounds under each of {full pre-diffusion, forced-blended, adversarial-withholding} to get tail estimates. The report's rare-event probabilities (14%, 31%) need substantial sample sizes to validate.

Two limits you can't get past

  1. Tail estimation. The report's most stressed predictions are p99/p99.9 events. A week-long testnet with ~100 nodes running Leios at 1 EB/20s gives ~30k EBs. That's plenty for p95, borderline for p99, thin for p99.9. Some conclusions in the report about rare-event certification failure will remain paper-only unless you run longer or with more nodes.

  2. Testnet vs mainnet mesh topology. Even if you match the number of nodes and regions, real SPO topology has network-of-network effects (relays behind stake pools, edge nodes on residential uplinks, community relays), that a controlled testnet doesn't reproduce. The blended-diffusion path distribution on mainnet may look quite different from a synthetic mesh. This is the same caveat as before but bites harder for blended-diffusion because path-length skew propagates through the convolution multiplicatively.

The short answer

DeltaQ traces give you (a) the single-hop CDF inputs to the report's convolution directly, (b) with modest additional per-EB arrival logging, the end-to-end multi-hop composed behavior, and (c) — most interestingly — exposure to dynamics the report doesn't model at all: inter-protocol contention, round-to-round feedback, recovery signatures. To exercise blended diffusion specifically, you also need a way to force pre-diffusion to fail on the testnet. The report's static-scalar α becomes a measurable time series once you're collecting traces this way, which is a meaningful qualitative upgrade to the analysis.

Clone this wiki locally