Skip to content

Performance Profiling

Jump to bottom Edit New page
github-actions[bot] edited this page Aug 28, 2026 · 78 revisions

Performance Profiling

How the throughput hotspots behind the numbers on the Performance page were found. This page documents the techniques and tools so future performance work (or a regression hunt) can follow the same playbook instead of rediscovering it. Everything here was used in the 2026 pass that took the HTTP/2 bridge arms from ~6× behind the managed reverse peer to parity-or-close.

Contents

The measurement harness

All throughput work starts from RpsLoadProbe:

  • Each arm is one topology: client protocol × origin protocol × TLS/cleartext × reverse/MITM (e.g. twp-reverse-http2-cleartext = H2 TLS client → H2→H1 bridge → cleartext H1 origin).
  • Every TWP arm has a control arm — the managed reverse peer (and the native reverse peer where it can run the path) hosting the identical workload in the same session, so both sides see the same machine state.
  • Every --ramp arm is three OS processes (parent load generator + origin child + proxy child) with a parent-seeded loopback CA (TWP_RPS_CERT_DIR). Combined --serve is debug-only. Absolute RPS from older combined TLS/QUIC-origin cells is not comparable to split runs — prefer TWP÷peer ratios.
  • The probe ramps concurrency (typically c=8→64) and reports sustainable RPS: the last concurrency step that still met the error-rate and p99-latency SLO. A ramp that grows RPS but blows p99 is a queue, not throughput.
  • Results land in timestamped CSVs under tools/RpsLoadProbe/results/. Publishable Performance tables cite GitHub Actions run IDs (median of 3 on matched 4 vCPU / 16 GiB runners). Cool paired A/B on a laptop proves a win before CI remasure; result tables live on Performance Local Lab.
# one suite
pwsh tools/RpsLoadProbe/run-rps.ps1 -Mode compare-bridges
# one arm, custom ramp (apphost, not `dotnet <dll>` — the child processes re-exec the host)
tools/RpsLoadProbe/bin/Release/net10.0/RpsLoadProbe.exe --ramp --mode reverse-http2-cleartext `
  --concurrency 8,16,32,64 --warmup-sec 2 --duration-sec 5 --results-dir tools/RpsLoadProbe/results

Controlling measurement noise

On a laptop, thermal throttling dominates everything else: the same arm measured 23k, 11k, and 51k RPS in one afternoon depending on accumulated heat. Rules that kept conclusions honest:

  • Compare only within one back-to-back session. Never compare a number from this run against a number from an hour ago.
  • Prefer TWP÷peer ratios over absolutes. The control arm soaks up the same throttling.
  • For a targeted A/B question, run the two arms paired: cooldown (~2 min idle), arm A, arm B immediately after — and alternate which goes first across repeats so heat bias cancels. This is how "MITM costs 0.65–0.75× of its reverse twin, and the delta is purely the extra origin TLS leg" was established: the two probe arms differ by exactly one flag (ForwardCleartext).
  • If an arm's ratio looks newly bad, re-measure before profiling — several "regressions" were heat.
  • Gate before publishing: optimize against cool paired ratios on the local lab. After a cool win, remasure on matched Windows+Linux GHA (workflow_dispatch RPS saturation) and paste medians into Performance. Windows GHA removes laptop thermal skew; it is not the same as a cool pair — shared-VM noise still applies, so prefer TWP÷peer ratios.

Laptop result tables

Laptop High-perf / cool-paired RPS tables and local saturation Blocks A/B/C live on Performance Local Lab. This page keeps the playbook only.

Technique 1: concurrency sweep as a shape test

Cheapest tool with the highest information density. Run the arm at c=1 and at c=64 and look at the shape, before reaching for any profiler:

Observation Meaning
Slow at c=1 and c=64 by the same factor Per-request cost (allocations, crypto, syscalls) — go CPU-profile it
Faster at c=1 but flatlines while the control arm scales A serialization point — something processes streams one at a time; profilers of per-request cost will mislead you
Scales to a cliff, then errors/SLO failures Resource exhaustion or a convoy (locks, pool limits, flow-control windows)

The h2c→H1 bridge showed the second shape: TWP beat the managed reverse peer at c=1 (6,425 vs 5,449 RPS) but flatlined at ~22k while the managed reverse peer scaled to 46k. That single observation eliminated allocation work, System.IO.Pipelines, and syscall efficiency as hypotheses and said "find the serial section."

Memory (RSS) — H2→H1 vs H1 / H3

Technique: connection-scoped Task retention + H2 bridge lite wire

Tools already on this page (cool A/B, concurrency sweep, dumpasync, stage timing) find where time burns. Memory gaps need the harness RSS sampler plus heap confirmation:

# Saturation / matrix CSV columns (every measure step):
#   proxy_rss_peak_bytes, proxy_cpu_avg_pct
pwsh tools/RpsLoadProbe/run-rps.ps1 -Mode compare-saturation

# Optional heap confirm under H2→H1 load (Full dump):
dotnet-gcdump collect -p <proxy PID>
# or: dotnet-dump collect -p <proxy PID> --type Full → dumpheap -stat

Root causes (H2→H1 ≫ YARP Memory while RPS ≈ parity):

  1. PendingSynthetics / PendingFinalizations retained completed Tasks for the life of each client H2 connection (ConcurrentBag never removed). Fixed with Http2PendingWork (remove-on-complete tracker) — connection-scoped retention, not a process-immortal leak.
  2. Per-stream SessionEventArgs on H2→H1 / H2→H3 even when IsFastPath (H1 keep-alive uses ResetForKeepAlive; H3 inbound already had H3H2FastForward). Mitigated with warm TryRentPooled + HeaderBuilder wire on interception-off bodiless H2→H1 (H3→H1 analogue) and IsFastPath skips on H2→H3.
  3. Custom H2 stack per client connection vs Kestrel pooled streams (residual).
  4. H3→H1 residual also includes MsQuic / QUIC connection pool cost vs YARP’s Kestrel QUIC stack — not a tiny session-lite leftover.
  5. Not the 768 KiB flow-control windows — advertised credit, not tiny-GET buffers. Do not force single H2 connection (or other single-connection env knobs) to game RSS (laptop A/B: −14% RSS / −23% RPS).

Keep / revert for new lites: cool-measure Memory + ÷YARP RPS; revert the lite if Memory is no better (within noise) or RPS regresses. Bag tracker is RPS-neutral retention — keep regardless.

Residual after bag + lite + ClientSyntheticStreams (GHA compare-saturation @ 571b6fba — 32724323848; same numbers in Saturation control Blocks B/C):

Arm OS TWP RSS YARP RSS ÷YARP
H2→H1 (twp-reverse-http2-cleartext) Windows 95 MiB 92 MiB ~1.04×
H2→H1 Linux 119 MiB 120 MiB ~0.99×
H3→H1 (twp-reverse-http3-cleartext) Windows 169 MiB 156 MiB ~1.08×
H3→H1 Linux 269 MiB 182 MiB ~1.48×

Pre-PendingWork saturation was ~5–9× YARP on H2→H1 (~848 / ~626 MiB). Pre-syntheticStreams Block B was ~1.6× / ~1.9×. RPS still leads YARP on both arms.

Decode-time SessionEventArgs skip (tried → reverted): H3-class defer of sessionFactory() until after HPACK on NullOrigin bridges (lite GET/HEAD/DELETE/OPTIONS + END_STREAM → Request bag + Response-only emit). Cool paired A/B after dispatch-order fix: ÷YARP RPS ~1.05× (no regression) but Memory flat vs prior lite-wire residual (~211 MiB TWP vs ~127 MiB YARP on laptop ≈ 1.65×, same band as CI ~1.6× / local lab ~199 MiB). First dispatch shape that awaited prior stream before starting the lite task capped multiplex (~0.5× YARP) — fixed before the keep decision. Reverted per keep/revert (Memory not past noise).

Windows Memory audit (2026-08-24) — synthetic-stream registry leak

Systematic Windows reverse inventory (published TWP RSS > YARP) ranked H2→H1 arms worst (~2.1–2.2×). Cool dotnet-gcdump / dumpheap on reverse-http2-cleartext @ c=64 showed:

  • RSS gap mostly outside managed GC heap (~244 MiB RSS vs ~17 MiB GC for TWP; YARP ~125 / ~5 MiB).
  • Actionable managed leak: ConcurrentDictionary<int, byte>+Node ~225k entries / ~14 MiB — syntheticStreams in Http2Helper.CopyHttp2FrameAsync TryAdd on every NullOrigin / synthetic stream and never removed on keep-alive multiplex (stream ids grow forever).

Fix (kept): wire Http2ConnectionState.ClientSyntheticStreams and clear via TryTakeStream / RemoveStream on all stream-exit paths (H2 helper, H2→H1 / H2→H3 bridges).

Cool laptop A/B after fix (mem-audit-post-synth-fix, c=64):

Arm TWP RSS YARP RSS ÷YARP Mem ÷YARP RPS
H2 TLS→H1 132 MiB 114 MiB ~1.16× ~1.20× (41.3k / 34.5k)
h2c→H1 105 MiB 93 MiB ~1.13× ~1.07× (47.6k / 44.6k)
H2 TLS→H3 139 MiB 124 MiB ~1.12× ~1.26× (20.3k / 16.1k)
h2c→H3 113 MiB 119 MiB ~0.95× ~1.10× (21.1k / 19.1k)

GC heap post-fix ~5.4 MiB; VolatileNode<int,byte>[] collapsed to one table (~0.6 MiB) with no Node storm. GHA paste @ 571b6fba: Win H2→H1 Memory ÷YARP ~1.04× (95 / 92 MiB); Linux 0.99×. H2→H3 / h2c→H3 cool remasure ≤1.2× Memory ÷YARP — no further dig on those arms (same keep).

Harness: tools/RpsLoadProbe/profile-memory-arm.ps1 (mid-measure gcdump + Heap dump). Linux: Dockerfile.mem-profile + profile-memory-arm.sh in Docker with libmsquic (diagnosis only — publishable numbers stay GHA).

H3→H1 cool remasure (same session, mem-audit-h3h1-post-synth): TWP 183 MiB / YARP 169 MiB ≈ 1.08× Memory ÷YARP (RPS ~10.2k / ~6.4k — YARP soft on this box). GC heaps ~10 / ~9 MiB; no unbounded dict Node storm. Published CI was ~1.57× — laptop gap is already under the ≥1.3× dig gate; residual treated as MsQuic / structural. No H3 Memory code change this pass; confirm with GHA Block C paste.

Unjustified Memory dig gate (post-syntheticStreams)

Skip treating Memory alone. From published reverse matrices, dig only when:

  • Memory÷YARP > 1.05, and
  • Memory÷YARP > RPS÷YARP (RSS tax without matching throughput lead, or RPS behind).

Otherwise Memory is at parity or justified by higher RPS. Do not close gaps as “YARP/Kestrel is leaner” — both are managed C#; attribute TWP-owned retention / pool / buffer causes.

Pri OS Arm Mem÷YARP RPS÷YARP Dig?
P0 Linux H3→H1 1.49× 1.15× Yes — largest excess
P0 Windows H3→H1 1.09× 0.95× Yes — more RSS and slower
P1 Linux H3→H2 1.28× 1.14× Yes
P1 Windows H1→H2 1.17× 1.03× Yes
P2 Windows H3→H2 1.20× 1.14× Marginal; after P0/P1

Out of scope when Mem ≤1.05 or Mem ≤ RPS: H2→H1, h2c→H1, H2→H3, h2c→H3, H3→H3, same-protocol H2↔H2.

# Windows mid-measure dump
pwsh tools/RpsLoadProbe/profile-memory-arm.ps1 -Mode reverse-http3-cleartext -Concurrency 64 -SkipBuild

# Linux (Docker + libmsquic) — build once, then:
docker build -f tools/RpsLoadProbe/Dockerfile.mem-profile -t twp-mem-linux tools/RpsLoadProbe
docker run --rm --entrypoint /bin/bash --cap-add=SYS_PTRACE --security-opt seccomp=unconfined `
  -v ${PWD}:/src -w /src -v ${PWD}/tools/RpsLoadProbe/results:/out `
  twp-mem-linux -c "sed -i 's/\r$//' /src/tools/RpsLoadProbe/profile-memory-arm.sh && bash /src/tools/RpsLoadProbe/profile-memory-arm.sh reverse-http3-cleartext"

KEEP — inbound H3 ConcurrentBag<Task> → Http2PendingWork: same defect class as the H2 bag. Mid-measure Win H3→H1 GC ~13 MiB with large Task[] segments → post-fix 3.4 MiB / peak RSS ~108 MiB vs YARP ~162 (0.67× Memory ÷YARP; RPS soft vs YARP on that cool pair). Linux Docker (diagnosis only): TWP ~142 / YARP ~209 MiB. Control-stream ReturnPayload also landed. Cool secondary after keep: H3→H2 Mem ÷YARP ~0.72×; H1→H2 Mem ~1.09× but RPS ~1.51× → skip dig. GHA @ f9769503: sat Block C Win 0.79× Mem / 1.12× RPS; Linux 0.77× / 1.07× — KEEP. Bridges H3→H1 Win 0.69× Mem / 1.02× RPS.

Remaining (Mem÷YARP > 1.05 and Mem > RPS) after that KEEP:

Pri OS Arm Mem÷YARP RPS÷YARP Notes
P1 Win H1→H2 1.12× 1.02× Extra origin TLS+H2 sessions vs YARP’s one; Offer-only-when-empty reverted (cool RPS 0.74× YARP) — next: skip live seed when pool already has capacity, not discard all seeds
P1 Lin H1→H2 1.08× 1.03× Same shape
P2 Win h2c→H1 1.10× 1.07× Soft; re-check after H1→H2
P2 Win h2c→H3 1.09× 1.04× Soft
P2 Win H2→H1 1.10× 1.05–1.14× Sat Mem>RPS; bridges Mem≤RPS — watch

RPS not beating YARP: Win H1→H3 ~1.00× (Memory already 0.89×). H3→H3 TWP not in compare-bridges (YARP control only) — prior paste.

nginx portable takeaways

Dig through nginx src/http (proxy + upstream keepalive), src/event, and src/core for reverse-proxy ideas that travel to a managed single-process proxy. Named mappings only — do not port the process model.

nginx idea Where TWP analogue Now?
Upstream keepalive cache (keepalive N → max_cached) ngx_http_upstream_keepalive_module.c MaxCachedConnections / MaxCachedConnectionsPerHost (already have) Done — no further cut
proxy_buffering on/off (buffer vs non-buffered upstream→client) ngx_http_upstream.c (u->buffering, non-buffered handlers) Sequential half-duplex session vs duplex pumps (StreamCopier-style / WebSocket dual copy / bridge body channels) Already shaped — see TWP vs YARP IO model; no new toggle
Write coalesce + sendfile / writev chain ngx_output_chain.c, ngx_writev_chain.c (“coalesce the neighbouring bufs”) Http2FrameWriter coalesce budget (already have); H1 is already buffered HttpStream writes Done for H2; H1 not a syscall-storm residual
worker_processes + accept_mutex nginx.c / ngx_event_accept.c N/A — native multi-process fan-out; TWP is one managed process + thread pool N/A — do not fake workers for RSS/RPS games

Concrete managed cut worth trying now: N/A for new session-lites. Portable lessons are already landed or process-model only. Decode-time H2 session skip was reverted; the Windows Memory audit then found and kept the syntheticStreams registry leak fix (see Memory (RSS)). Next dig is residual native RSS / MsQuic after GHA remasure.

Technique 2: async dumps — find where requests wait

CPU profilers show where cycles burn; under async I/O the bottleneck is usually where requests park. Capture the async state machine population under load:

# must be a Full dump; a mini dump lacks the heap metadata dumpasync needs
dotnet-dump collect -p <proxy PID> --type Full
dotnet-dump analyze <dump file>
> dumpasync --stats

Read the histogram of parked continuations. In the bridge investigation, hundreds of in-flight requests were parked in synthetic-response emission waiting on one SemaphoreSlim (the client write lock) — a classic convoy, with the side signature of high system CPU from many tiny socket writes. The fix (a dedicated per-direction frame writer draining a channel and coalescing up to 32 frames / 32 KB per socket write, Http2FrameWriter) was worth 3.4× on that arm.

Technique 3: per-stage latency decomposition

When internal work looks fast but clients still see high latency, decompose the request path. TWP already captures per-request milestones when EnableRequestTimingCapture is set (see Request timing); RpsLoadProbe has an opt-in collector that aggregates them under load:

# any non-empty value enables; a path (length > 1) writes reports to that file
$env:TWP_RPS_STAGE_TIMING = "C:\temp\stage-timing.log"

StageTimingCollector subscribes to AfterResponse, buckets HttpRequestTiming durations (client read, connection wait, request send, TTFB, delivery, total), and prints p50/p90/p99 per stage every 20 s. Subscribing to AfterResponse disables the no-interception fast path, so this must stay out of publishable runs.

The decisive read: the internal pipeline showed p50 87 µs per request while clients observed p50 2.6 ms — so ~2.5 ms of queueing happened before a stream entered the instrumented pipeline. That pointed at the per-connection HTTP/2 frame loop, which was running each stream's BeforeRequest handler prefix inline (~44 µs per HEADERS frame), capping any single client connection at ~22k streams/s regardless of concurrency. Starting the handler on the thread pool from the already-ordered dispatch task took the arm from 22k to 47k RPS.

The same collector separates "proxy is slow" from "origin leg is slow": on the H1→H2 bridge, TTFB was 240 µs at c=8 but 1,830 µs at c=64 with barely more RPS — the signature of CPU saturation, not another serial section.

Technique 4: CPU sampling

For arms where the sweep says "per-request cost" or "saturation," attach the sampler during a long window (the ramp's default 5–7 s steps are too short to attach; use a 150 s single-concurrency run):

tools/RpsLoadProbe/bin/Release/net10.0/RpsLoadProbe.exe --ramp --mode <arm> --concurrency 64 `
  --warmup-sec 2 --duration-sec 150 --results-dir tools/RpsLoadProbe/results/profiling
# ramp logs print: attach: split origin pid=… proxy pid=…
# in a second shell:
dotnet-dump collect -p <proxy PID> --type Full
dotnet-trace collect -p <proxy PID> --profile dotnet-sampled-thread-time --duration 00:00:25

This is a confirmation tool more than a discovery tool here: it confirmed the residual H1→H2 gap after origin-connection sharing is still whole-box cost (dual TLS legs plus the per-request session pipeline). Sharing lifted the arm from 0.33× to 0.53× peer at c=32 (rps-ramp-20260818-130040 / 130112); cool remeasure after grow-at-4 stayed ~0.51× (profile-baseline / profile-post-fix). TTFB still rises with concurrency. At c=32 dumpasync showed 8 origin ReadLoopAsync instances (pool already spreading) plus Monitor / SslStream in the sampled stacks — not a single-conn convoy. Honest remainder: dual-TLS + session cost on this 8-thread box.

Technique 5: reference-source comparison

When a comparable managed reverse peer is faster, read its source to answer named hypotheses — not to port its architecture. Two examples from this pass:

  • "Does the reference .NET server stack tune MAX_CONCURRENT_STREAMS dynamically?" No — it opens additional origin connections when the stream limit is hit. TWP replicated the behavior within its own design (Http2OriginRelayPool).
  • "Is System.IO.Pipelines the advantage?" No — TWP's buffered HttpStream already amortizes socket reads to one syscall per buffer drain; the memcpy ReadOnlySequence would remove costs ~0.02% of a request, and the TLS decrypt copy exists in both models (SslStream cannot produce a ReadOnlySequence; the reference .NET server stack copies decrypted bytes into its Pipe too). Measured support: H1 arms at parity, and TWP's c=1 latency lower than the managed reverse peer's. The layering difference is real; it is not why the tiny-GET tables look the way they do — see TWP vs YARP IO model.
  • "When does the managed reverse peer open another origin H2 connection?" ForwarderHttpClientFactory sets EnableMultipleHttp2Connections = true by default — SocketsHttpHandler grows sessions under stream pressure. TWP's PoolGrowActiveStreamThreshold is the analogous dial (lowered 16→4 after profiling).

TWP vs YARP IO model

Architecture context for future gap hunts — not a root-cause claim. YARP sits between Kestrel (System.IO.Pipelines on the inbound connection) and SocketsHttpHandler, with HttpForwarder / StreamCopier able to run request and response pumps at once. TWP's classic HTTP/1 session is one async state machine: send request, copy request body, wait for response, copy response body (HandleHttpSessionRequest / CopyBodyAsync). That is leaner for half-duplex HTTP, which is most of the RPS probe.

Do not over-count the tax:

  • Kestrel's transport read/write loops are mostly per connection, not four new tasks per tiny GET. Concurrent copiers matter when there is a body or true duplex.
  • Extra Tasks are extra continuations, not extra OS threads. Both stacks yield on I/O.
  • TWP H2/H3 is already multi-task (frame intake, Http2FrameWriter, per-stream dispatch, origin read loops). WebSockets already run two CopyToAsync relays (TcpHelper.SendRawTap). The "one worker" story is H1 request/response, not the whole product.
  • A pipe's real win is a bounded buffer between stages (writer pauses when the reader is slow). await WriteAsync in a copy loop already stalls the next read. TWP already has BoundedBodyPipe and bounded H2→H1 DATA channels (FullMode = Wait).

YARP inherits better insurance for slow consumers, protocol-edge bridging, and full-duplex (gRPC / streaming POST / response-starts-early) because those behaviors come with Kestrel + the forwarder. That is a workload preference, not a ceiling. Match it locally without becoming YARP:

Workload Do Do not
Tiny-GET / H1 POST Keep the sequential session. It already wins or ties. Put a pipe + two copiers under every request.
Slow consumer / lossy H1 Bound the origin→client copy; keep frame-writer coalesce. Blame "missing Kestrel" before measuring buffer/backpressure.
Protocol bridges Keep the direct bridges; fix unfinished paths (e.g. H3 POST sustain 0). Port Kestrel for speed — several bridges already lead YARP.
True duplex Start both pumps while both directions are live (same pattern as WebSockets / H2 compressed-relay send+receive). Force one linear H1-style task to own a bidirectional stream; rewrite H2↔H2 as Kestrel+StreamCopier for one cell.

Remaining YARP-led cells in this pass (H2 POST at c=32, some 256 KiB bodies, lossy H1) were named as multiplex/shared-writer, copy/syscall/coalesce, or buffer-vs-delay — not "we lack pipelines." nginx still leads both on Linux H1; if there is a hard ceiling it is managed C# vs native, not TWP vs pipes.

The architecture-sensitive laptop table (and CI medians on Performance) is compare-arch. Duplex H2 remains YARP-led by design: interception-off H2↔H2 already runs concurrent compressed frame relays (CopyHttp2FrameAsync both directions) — not the sequential H1 session. Published CI: Win ≈ 0.63× (1,670 / 2,666), Linux ≈ 0.31× (579 / 1,862) @ 1f2d0eee. Low TWP CPU on that cell (Win ~23% / Linux ~13% vs YARP ~41% / ~52%) is the concurrent-copier insurance gap, not a missing H3-style WhenAll(upload, ReceiveResponse) cut. Early-response H2→H1 (TWP leads) and WebSocket dual-copy (TWP leads) do not imply H2↔H2 duplex will match YARP without adopting that forwarder model.

Case studies: symptom → tool → root cause → fix

Symptom Tool that found it Root cause Fix
h2c→H1 bridge 8.5k vs peer 46k, high system CPU dotnet-dump + dumpasync --stats Response emission convoy on the client write lock; many tiny socket writes Queue all response frames through Http2FrameWriter (coalesced writes) — 3.4×
Same arm flat at ~22k at every concurrency, but faster than peer at c=1 Concurrency sweep + stage timing (87 µs internal vs 2.6 ms observed) Frame loop ran each stream's BeforeRequest prefix inline (~44 µs/HEADERS) Start the handler on the thread pool from the ordered dispatch task — 22k → 47k
External-site H2 downloads stalled at exactly 64 KB Standalone repro tool (tools/H2ExternalRepro) + a window-size env knob Flow-control starvation: batched WINDOW_UPDATE threshold larger than the default 65,535 B window Advertise a reference .NET server stack-class 768 KiB initial stream window in both directions
Two bridge arms at 100% errors after the passthrough change The benchmark suite itself (error-rate SLO) :scheme mismatch in compressed header relay on mixed-transport bridges Detect and re-encode the header block with the corrected scheme
POST arm collapsed 842 → 9 RPS Benchmark suite + targeted repro Client DATA frames raced the handler dispatch and were routed before channels existed Await the stream's dispatch task before routing its DATA frames
H1→H2 bridge stuck at ~0.3× peer Stage timing (TTFB 240 µs → 1,830 µs as c grows) + dotnet-trace Dual TLS + per-request pipeline; also one dedicated origin H2 connection per H1 client Shared Http2OriginConnectionPool (0.33× → 0.53× at c=32). Remainder still looks CPU-bound
H3→H2 SLO-failed above c=16 (then 100% errors after pooling) Error log (TWP_H3_ERROR_LOG) + RFC 7540 §5.1.1 Exclusive ConcurrentBag cap 16, then concurrent SendAsync allocated stream ids off the write lock so a higher id's HEADERS could hit the wire first; the reference .NET server stack implicitly closed the lower idle streams and GOAWAYed Shared pool (no exclusive checkout) + allocate stream id and write opening HEADERS under the same write lock. Reverse H3→H2 holds c=64 at 0% errors (rps-ramp-20260818-130231)
Inbound H3 ~0.40× peer blamed on “managed QUIC vs MsQuic” Code read of QuicClientHandler.ListenQuic Inbound H3 already is System.Net.Quic / MsQuic. Pre-match ratios also mixed quic-http3 vs HttpClient Matched-client H3→H1 ≈ 0.87; do not prototype a second QUIC stack
H1→H2 / H3→H2 still ≪0.80 after pool Cool A/B + c=1 + dumpasync/dotnet-trace @ c=32 (results/h2-origin-choke/) Not dual-TLS polish: c=1 TWP faster (1.49×). Residual is outbound Http2OriginConnection.SendAsync queueing (TTFB≈SendAsync wait grows 624→2263 µs c=8→32; 13 parked SendAsync on H3→H2; 102 SemaphoreSlim TaskNodes on H1→H2; managed reverse peer only ~7 in-flight forwarders). Monitor slow-path ~2× managed reverse peer Origin Http2FrameWriter exclusive drain: encode+enqueue under short writeLock, no WriteAsync under the lock (reference .NET server stack model). Cool H1→H2 0.87× @ c=32 (28,996 / 33,336, rps-ramp-20260818-170412/170452); H3→H2 0.64× @ c=32. TTFB p50 262→894 µs. See h2-origin-choke/POSTFIX.md
H1→H2 / H3→H2 still <0.80 after origin frame writer Cool A/B + grow A/B + gcdump/trace (results/residual-sub08/) Scaling wait on origin HEADERS (c=1 1.04×); grow 4→32 regresses; ForceRead/HPACK noise; Channel/Pipe not retained-heap Ranked in residual-sub08/CONCLUSIONS.md
Monitor.Enter_Slowpath ~9.5% after frame writer syncblk + speedscope + pool-pick diag; post-fix traces (POSTFIX.md) ~70% Monitor was TryPick + ConcurrentDictionary.Count under entry.Gate. c=32: 0% soft-miss. c=64: ~21% soft-miss + CreationGate at max Shipped A+B+C: Interlocked ActiveStreamCount; skip CreationGate on Gate-held Count >= max; snapshot pick outside Gate. Monitor exclusive 9.5% → 3.1%. Phase C no further win. Long-window TWP @ c=32 unchanged (~28.7k). Residual still HEADERS fan-in
H1→H2 still ~0.71× after pool-pick; dumpasync showed ForceRead on origin ReadLoop Cool remasure + code path (POSTFIX-INTAKE.md) Origin ReadLoop still did ForceRead 9+payload and copied HEADERS to MemoryStream; DATA awaited BodyPipe on the loop Shared Http2FrameIntake on origin + in-place END_HEADERS decode + sync BodyPipe write. ForceRead removed. Best long cool pair this session still 0.71× (thermally soft absolutes) — next dig is post-headers path, not another receive rewrite
Post-intake: is residual WriteResponse / SessionEventArgs / still HEADERS wait? dumpasync + topN + gcdump + stage timing (POSTFIX-POST-HEADERS.md) Soft box (IDE CPU); dumpasync: no ForceRead, no InterimChannel/SendAsync park — bridges on client ReadRequestLine, origin on FrameIntake.Fill. Stage: TTFB ~93% of total, delivery ~5%. Pooling gates not cleared Wait shape fixed. Do not pool or rewrite H1 write yet. Need cool quiet remasure + high-RPS alloc/CPU sample before next code change
Quiet remasure after restart: does cool ratio move? Gate A/B at high RPS? Cool pairs + dumpasync + AllocationTick (quiet-remeasure/QUIET-REMEASURE.md) High perf: H1→H2 c=32 0.71× (31.9k/44.7k); c=64 0.87×. High-RPS dump: ForceRead/Interim park still 0. AllocTick: SessionEventArgs+HeaderCollection 4.5% (<5% Gate A). Interim channel arrays ~7%+ but gated behind A. Monitor exclusive ~2.5% No library change. c=32 residual confirmed; pooling/write gates still not cleared. Optional: YARP twin AllocTick for asymmetry
YARP twin AllocTick + InterimChannel passthrough lite Twin gc-verbose + remasure (INTERIM-LITE.md) TWP ~3× AllocTicks/request vs YARP; Interim Channel/segment ~11% TWP-only. Lazy InterimChannel when on1xx null; H1→H2 passthrough skips relay when no interception Landed. Soft post-lite pair 0.82× (26.8k/32.5k); cool High-perf confirm blocked by IDE CPU — remeasure on quiet box before publishing ≥0.80
Cool confirm after InterimChannel lite Paired c=32 High perf (interim-lite-confirm/CONFIRM.md) TWP 33.6k / YARP 44.0k = 0.76× (was 0.71× pre-lite). Phase-A-class absolutes Lite helped (~+5–7% relative) but still ≪0.80. Next: TTFB residual dig on no-intercept path
Post-lite TTFB dig @ ~31k RPS dumpasync --fields + topN (interim-lite-confirm/TTFB-DIG.md) 20 SendAsync on origin writeLock (Semaphore maxCount=1, streamOpened=false); 6 on HeadersReceived; InterimChannel still 0. Lite on1xx=null confirmed Residual is writeLock stream-open convoy, not headers wait / WriteResponse. Next: shrink work under origin writeLock (HPACK encode+enqueue)
H3→H2 cool remasure + gap fix plan Cool c=32 pair (h3h2-fresh/CONFIRM.md) + FIX-PLAN.md / canvas H3→H2 0.70× (26.0k/36.9k) — wiki 0.33× stale. Same origin writeLock; H3 still always allocates InterimChannel P0 H3 Interim lite → P1 shrink encode under writeLock → P2 H3 Via/prep trim → P3 remasure other H2 arms
P0+P1 bundle: H3 lite, Via skip, SoftStream=2, HPACK method cache Cool High perf (post-p0p1/) H1→H2 0.89× (37.3k/42.0k); soft confirm 0.82×. H3→H2 0.73× (24.9k/34.2k). Max-conn 16 aborted (soft regress) H1→H2 c=32 bar closed. Continue H3→H2 (≥0.80) + remasure other H2 arms
Remeasure H2 TLS→h2c / h2c→h2c after intake+lite era Cool High perf c=32 20s (passthrough-fresh/) H2 TLS→h2c 0.78× (51.0k/65.8k); h2c→h2c 0.73× (49.7k/68.3k) — up from ~0.66/0.70 wiki Still ≪0.80 on passthrough; next dig client FrameWriter/HPACK (not origin pool)
HPACK static GetIndex bug + encode under writeLock + scheme patch Cool High perf (post-hpack-static/ + post-hpack-confirm/) StaticTable.GetIndex(name,value) compared ByteString to string → never matched; EncodeHeaderBlock allocated new Uri under writeLock; mixed-transport scheme 0x86↔0x87 patch; SoftStream=1; skip Via on H2 response IsFastPath; skip NoOp HPACK decode on verbatim compressed relay H2 TLS→h2c 0.81× (45.8k/56.2k) closed. H1→H2 0.85×. H3→H2 0.72×, h2c→h2c 0.74× still open
OriginRelayPool SoftCap 8→1/2 fan-out Cool remasure (post-relay-soft1/2) Soft=1/2 did not beat Soft≈8 on h2c→h2c (extra cleartext legs) Reverted SoftCap formula; residual is not origin-leg count
H3→H2 dump @ 26k RPS + QPACK dict encode dumpasync (h3-profile/) + QPACK O(1) static lookup 32/32 SendAsync on HeadersReceived (not writeLock); 8 origin ReadLoops. SoftStream fan-out already enough Residual is H3 session/QPACK/bridge CPU, not origin write convoy. QPACK static dict + response header path trim shipped; cool ratio still ≪0.80 — next SessionEventArgs-lite / pool
H3 inbound ≪ H2 / ≪ YARP H3→H3 Cool YARP-first matrix + shape (h3-vs-h2/, h3-verbatim-fair/) Full SessionEventArgs + response QPACK decode/re-encode on every H3→H3 GET; YARP cool H3→H3 ~26–28k while TWP sat ~20k (0.70×) Session-lite for H3→H2/H3/H1 + verbatim origin→client H3 frame relay (H2 compressed-relay analogue). Cool H3→H3 1.14× YARP (29.6k / 26.0k); H3→H2 / H3→H1 ≥0.80; MITM÷cleartext 0.93
H3 bodiless fast path + PrepareH2 skip + EncodeResponse + compressed DATA→wire Cool High perf (post-encode-response/) Skip InterceptionContext; drain FIN without body-pump lambdas; skip PrepareH2 RemoveHeader scan on IsFastPath; QpackEncoder.EncodeResponse (no List); compressed-relay DATA ReadExact into rented wire buffer; ReturnPayload after QPACK decode Absolutes up (H3→H2 31.7k/44.1k; h2c→h2c 65.7k/91.3k) but ratios still ~0.72×. Lazy BoundedBodyPipe aborted (empty-body race). Skip linked-CTS on fast path aborted (abort cancel lost → ~0.67×). Next: SessionEventArgs-lite / pool
H3→H2 SessionEventArgs-lite (H3H2FastForward) Cool High perf (h3-lite-only/) Skip entire session/HttpWebClient/Null stream/empty Response on interception-off bodiless H3→H2; keep Request for HPACK only H3→H2 0.83× (33.2k/40.0k) closed. Lazy BodyPipe re-tried + aborted again (empty-body hang / ~856 RPS). h2c→h2c still 0.74× (56.7k/76.3k)
h2c ThreadPool IOCP floor + SoftCap32 / exclusive drain / sync cont Cool High perf (h2c-iocp-min/, aborted siblings) Profile: LowLevelLifoSemaphore wait ~46%. Mirror worker min onto IOCP; default worker floor ×8/64. SoftCap32 / exclusive FrameWriter / sync continuations / WINDOW_UPDATE enqueue / CTS TryReset all aborted (regress or hang) h2c→h2c 0.76× (62.6k/82.5k). Still open vs ≥0.80
Compressed finalize sync + CTS TryReset (no SoftCap change) Cool pairs (h2c-cts-reset2/, h2c-soft16/) Skip PendingFinalizations Task on compressed; TryReset pooled CTS TWP absolutes up to ~65.7k on soft YARP; cool YARP-first still ~0.75×. SoftCap floor 16 neutral; SoftCap 32 convoy
Compressed END_STREAM skip force-flush WINDOW_UPDATE Cool sandwich (h2c-wu-batch-confirm/) Tiny-GET forced connection WINDOW_UPDATE pair per ~56 B response (~6% GrantReceiveCredit); batch connection credit to 384 KiB, drop stream credit on close h2c→h2c 1.20× (102.8k/85.9k) closed — TWP leads. TWP A/B both ~102k
Full reverse+MITM cool audit + scheme decode-free + H2→H1 bridge trim Cool High perf (gap-audit/, gap-audit-yarp/, h2h1-post-cts/, h2h1-vs-h2h2/) Wiki h2c→H2 TLS 0.57× / MITM H3→H2 3.5k stale. Mixed-transport :scheme skips Decoder when 0x86↔0x87 patch works. H2→H1: buffer tiny bodies, skip Linked CTS / framing validator / empty Before* on IsFastPath; struct enumerator for LowercaseHeaderNames. New reverse-http1-mitm fair twin h2c→H2 TLS 1.29× (89k/69k). H1 plain 1.19×. H1 TLS 0.91×. Transparent H1 MITM÷terminate 0.96×. H2→H2 MITM 104k. H2→H1 MITM÷cleartext 0.82–0.86× residual (per-stream H1 fan-out × dual TLS; H2→H2 dual-TLS proves TLS not the choke)
Origin HTTPS: unwrap nested HttpServerStream + skip SslStream header flush Cool (ssl-unwrap2/) + dump diagnosis Direct HTTPS was NetworkStream→HttpServerStream→SslStream→HttpServerStream; header Write flushed every SslStream. Finish() no-op when Content-Length exact Correctness/cleanup landed; cool MITM÷cleartext still ~0.84× (47.6k/56.8k). Residual not nesting/flush
H2→H1 MITM÷cleartext ≪0.90 after ssl-unwrap Cool 3/5-rep High-perf (h2h1-hostcache/, h2h1-gate12/, h2h1-final/, h2h1-harden/) + thread-time sample Whole-RT SoftCap serialized warm SslStream reuse; DataAvailable closed keep-alives after exact-CL reads; per-request pool-key StringBuilder; cert callback .Wait on CompletedTask Create-only SoftCap (MaxConcurrentHttp11HttpsOriginCreates, default Clamp(ProcessorCount,4,32)); always close on residual DataAvailable (no IsFastPath skip); cache CachedHttp11PoolKey; sync cert validation. Cool 5-rep median 0.90 / mean 0.92 (h2h1-final/); harden 3-rep median 1.03 / mean 0.91 (h2h1-harden/). Lock-free pool rent aborted (noisy dips)
H1→H2 still ≪0.80 after pool + named session micro-opts Cool matched A/B (matched-post-fix) Dual client+origin TLS + per-request session on an 8-thread box; dumpasync already showed multiple origin ReadLoops (not a single-conn convoy) Superseded by h2-origin-choke/ (2026-08-18): see row above
H3→H2 c=8/16 lost ~30–40% vs exclusive-bag after pooling Cool A/B (profile-baseline) + dumpasync/dotnet-trace @ c=16 (h3h2-c16.dmp / .nettrace) Grow threshold 16 pinned all streams on one origin ReadLoopAsync; 716 SemaphoreSlim waiters; H3 GET also did HEADERS + empty DATA Grow at 4 active streams + drain FIN then HEADERS+END_STREAM for bodiless H3. Recovered 8,418 @ c=16 (profile-post-fix, vs phase-0 8,539)
H2 TLS→h2c / h2c→h2c ~0.63–0.66× cool dumpasync + sampled trace @ c=32 Http2FrameWriter already on DATA path; ForceRead per frame header; HEADERS still two WriteAsync under the lock Large-read Http2FrameIntake (64 KiB) + enqueue stream-scoped HEADERS on Http2FrameWriter. Matched cool h2c→h2c ≈ 0.70, H2 TLS→h2c ≈ 0.66 (matched-post-headers-writer)
Cool H3→H1 ~0.36× peer (12.1k / 33.4k) Cool pair + trace @ c=32 Invalid ratio: TWP quic-http3 vs peer HttpClient. Trace was session/HandleAsync, not MsQuic-native Match clients; later dual-listen reverse H3 enables HttpClient both sides (matched-httpclient-h3/, H3→H1 ≈ 0.87)
H3→H1 integ empty body; Windows ~0.79× YARP DualListen / ForcedHttp11Origin + cool pair Fast path buffered only known Content-Length; Kestrel WriteAsync often chunked → body never drained before pool Release; H1 Title-Case names paid QPACK ToLower every response Drain chunked/connection-close via LimitedStream before Release; NormalizeNamesToLowerAscii + HeaderNamesAreHttp2Normalized; decode H2 HEADERS into Response headers (no second collection). Cool Windows H3→H1 ≈ 0.96×, H3→H2 ≈ 1.06×; Linux H3→H1/H2/H3 ≈ 1.04× / 1.15× / 1.20× (32552296839, 32552295495)
H1→H3 “name normalize + tiny-body coalesce” looked like H1→H2 gap Cool A/B (win-parity-audit-20260822-*) Hypothesis: Title-Case QPACK tax + missing H1→H2-style fast commit / TLS coalesce Name-normalize path fully reverted (~1.13× → 0.65×). Eager-buffer alone also poisoned the pool — see next row.
H1→H3 Win CI ~0.94×; H1 client + StreamBodyWriter = header-only TLS record Cool A/B + dispose/RST dig (cool-h3-origin-eager64-drain-20260823/) Known-CL ≤64 KiB H3 origin bodies streamed via StreamBodyWriter → H1 WriteResponseAsync then body (same class as lossy H1). First eager-buffer attempt disposed the Quic stream before FIN → RST / pool poison (1.16×→0.7×) Eager-buffer ≤64 KiB and drain frames to FIN before DisposeAsync in ForwardOverQuicAsync. Cool TY/YT ≈ 1.23× / 1.03×. CI remasure bridges next.
H2→H1 64 KiB ~0.87× YARP (tiny-GET already parity) Cool pair + code compare vs Kestrel/YARP Streamed path stripped Content-Length then empty END_STREAM DATA; pump wrote 8 KiB fills → 8 DATA frames + trailer; HttpStream double-buffered socket→8 KiB→dest; QueueDataFrame + 32 KiB flatten Keep CL + END_STREAM on last DATA; HttpStream large-read bypass; in-place DATA framing (flatten kept); skip LimitedStream/Via on known-CL fast path; raise flatten budget to 288 KiB. Cool 64 KiB ≈ 1.13×; 256 KiB ≈ 0.89×. Dropping flatten alone still ~0.65×
H2 POST cool ~0.88× / 256 KiB H2→H1 ~0.90× Shape c=1 vs c=32 + YARP StreamCopier (64 KiB) compare c=1 TWP leads POST (~1.2×); c=32 loses when YARP healthy — multiplex tax (frame-loop copy + shared client writer). Extra body memcpy / coalesce experiments Kept: ArrayPool request-body channel + TryReserve on CopyFromAsync. Do not: reserve >1 frame before enqueue; slice control frames into coalesced DATA; drop flatten
H3 early-response Win CI ~0.76× (Linux already ~1.02×) Cool A/B (fix-early-tls/) + origin/YARP duplex compare ForwardOverTcpAsync wrote the full request body before ReceiveResponse while the probe origin overlaps after 8 KiB (YARP StreamCopier same). H3+MsQuic amplifies the serialization on Windows Overlap streamed upload with ReceiveResponse; fold remaining upload into StreamBodyWriter via Task.WhenAll. Cool mean ≈ 1.21× YARP. Do not re-land Http3Frame coalesce 256→16 KiB (hurt POST)
Duplex H2 Win CI ~0.63× / Linux ~0.31× (compare-arch) Code path + CI medians 32688089789; short local cool noisy H2 TLS↔H2 TLS overlapping 64 KiB POST. Interception-off reverse already concurrent-relays frames both ways (not H1 sequential; not the H3 pre-overlap ForwardOverTcpAsync bug). YARP HttpForwarder/StreamCopier + Kestrel still leads; TWP CPU underutilized vs YARP on the cell No product cut. Document as irreducible YARP-led concurrent-copier cell; keep published CI ratios. Do not port Kestrel/StreamCopier for this row alone.
H3→H1 64 KiB GET Win CI ~0.56× / Linux ~0.82× Cool A/B (h3-64k-rebaseline/) + CI remasure Cool mean ≈ 1.13× (3118/2688 & 3488/3181); stale CI was pre-StreamBodyWriter No library change. Publishable 32611185635 @ cd276c83: Win ≈ 1.15× (3,752 / 3,269), Linux ≈ 1.25× (5,295 / 4,247). Next body gap: Win H1 TLS 256 KiB ≈ 0.85×
H1 TLS→H1 256 KiB Win CI ~0.85× Cool A/B + shape (h1-256k-cool/) + YARP StreamCopier compare Cool c=1 ≈ 0.83× (per-request); CopyBytesToStream FillBuffer’d 8 KiB forever — H2 large-read bypass never ran on H1 known-CL copy Rent 64 KiB + ReadAsync when parser window empty (HttpStream.CopyBytesToStream, 106e73b9). Cool c=1 ≈ 1.16×, c=32 ≈ 1.09×. Publishable 32614286032: Win ≈ 1.12× (2,617 / 2,347).
H1 TLS new-conn Win CI ~0.84x (Linux TWP leads) Cool A/B + Kestrel SocketConnectionListener / ConnectionDispatcher + bare ceiling Nested SslStream + ClientHello peek + ECDSA + Task.Run + BeginAccept APM + per-accept linger/timeouts + RetryPolicy closures; lite path forwarded Connection: close to origin → no origin pool under NC Peek/unwrap/RSA/no-keepalive; abortive SO_LINGER(0) on close; AcceptAsync; CTS pool; session-lite; WaitForData-before-SslStream; 8 KiB rent. AcceptIOQueue no win. Bare NC Connection: close response-skip fixed. Strip hop-by-hop Connection before origin write on H1 terminate lite. Publishable 32625349927 @ 13059143: Win NC ≈ 1.01×, Linux NC ≈ 1.01× YARP (nginx 1st on Linux NC — TWP 2nd).
H1→H3 100% err after session-lite (03159694) Bisect 11e32f1c→03159694 + curl serve H1 terminate lite gated only on ForwardHost + bodiless GET — H1→H3/H2 with forced upstream H3/H2 took TCP H1 lite against QUIC/h2-only origins Skip session-lite when connection-level UpstreamHttpProtocol is Http2/Http3 (62e5efcd). Soft coolish H1→H3 ≈ 1.25×, h2c→H1 ≈ 1.05×, H3→H1 ≈ 1.09×.
WarmTls H1→H3 + CachedServerAuthOptions gate broke H2 reverse Local lossy H2 + ALPN fail (No common application protocol) Expanding fixed-cert to any warmed CachedServerAuthOptions pinned http/1.1-only ALPN while H2 clients offer h2 Gate fixed-cert on !EnableHttp2 only (H3 clients use QuicListener). H1→H3 host: EnableHttp2=false + WarmTls (8ac422ee). Cool H1→H3 ≈ 1.16×; remasure bridges/bodies/lossy @ tip.
H2/H3→H1 64 KiB / lossy H2 still tax many DATA fills Code compare vs H1 ≤64 KiB coalesce H2→H1 / H3→H1 eager-buffer capped at 16 KiB while lossy/bodies GET is 64 KiB → stream via ~4× 16 KiB fills (shim delayMs per read) Raise eager known-CL threshold to 64 KiB (Http2ToHttp11BridgeHandler + Http3OriginBridge, 8ac422ee). CI bodies Win H3→H1 64 KiB ≈ 1.09× (32631121563).
H3→H1 tiny Win CI ~0.90×; cool order-noisy ~0.94–1.02× Cool both-order dig + HEADERS+DATA coalesce experiment SendResponseAsync does separate QuicStream writes for HEADERS then DATA + Flush Reverted single-write HEADERS+DATA coalesce (cool-h3-headers-data-coalesce-20260823/): cool mean ≈ 0.96× (no win).
H3→H1 c=1 leads ~1.25×; c=32 loses (~0.96×) — multiplex shape Cool c=1 both orders + dumpasync @ c=32 (dig-h3h1-dump2-20260823/) + SampleProfiler 32/32 parked on origin ReceiveResponse→FillBuffer (not writeLock/SoftCap). CPU: both TWP and YARP ~60% LowLevelLifoSemaphore wait — not a unique ThreadPool starve. QPACK/Normalize ≪1% exclusive Next: cycle-time after status line (body buffer → QPACK → Quic write → pool Release) vs YARP HttpForwarder; optional AllocTick asymmetry. Soft remasure @ 8e5c181b (cool-h3h1-shape-20260823/): c=1 ≈ 1.60×, c=32 ≈ 1.07× — CI Win still ~0.89× @ prior tip; remasure bridges in flight.
H3→H1 early origin Release / skip client Flush before CompleteWrites Cool A/B (cool-h3h1-early-release-20260823/, cool-h3h1-skip-flush-20260823/) Hypothesis: free H1 socket sooner / avoid MsQuic Flush tax on fast path No cool win (~0.95–0.98×). Reverted both.
H1→H3 / h2c→H3 Win CI closed @ 8789d6de CI bridges 32636039240 Eager ≤64 KiB H3 origin body + FIN drain before Dispose Win H1→H3 ≈ 1.04×, h2c→H3 ≈ 1.05×. Still open: Win h2c→H1 ≈ 0.96×, H3→H1 ≈ 0.89×; lossy H2 16 vs 17.
H3→H1 ForwardHost Host rewrite (match YARP HttpForwarder) Cool A/B (cool-h3h1-host-rewrite-20260823/) Hypothesis: :authority localhost:<listen> on H1 Host wire vs origin 127.0.0.1:<port> No cool win (~0.94–0.99×). Reverted.
h2c→H1 early origin Release before EmitSynthetic Cool lead ~1.03×; CI remasure 32638840153 @ 253e8716 Hypothesis: free H1 socket before H2 frame emit CI miss: Win h2c→H1 still ~0.95×; Lin h2c→H1 regressed ~1.03×→0.96×. Reverted.
h2c→H1 ForwardHost Host rewrite Cool A/B (cool-h2ch1-fwdhost2-20260823/) Same Host rewrite on H2→H1 bridge No clear cool win (YT ~0.97×). Reverted.
Lossy H1 Win remasure @ 253e8716 ≈ 1.00× (662/662) CI lossy 32638842839 Prior ~0.99× noise Closed ≥1.00× (nginx 1st 634 — TWP 2nd). Win lossy H2 still 16 vs 17.
Lossy H2 HOL: NullOriginStream lacked SETTINGS_MAX_CONCURRENT_STREAMS Cool + CI lossy 32643126466; bridges tax @ same tip Hardcoding MaxStreams=8 on NullOriginStream closed lossy (Win 3.47×) but tax'd Win tiny-GET h2c→H1 / H2→H1 (~0.90×) via extra TCP handshakes at c=64. Landed lossy-only: probe sets ResourceLimits.MaxConcurrentStreamsPerConnection=8 when IsLossy (WithMaxConcurrentStreams…); Http2Helper appends SETTINGS. Tiny-GET keeps default 256. Cool lossy 29/15; cool h2c ≈ 1.04×. Remasure lossy+bridges. Dispose harden on TcpClientConnection kept.
Lossy H1 Win cool ~0.86× (p50 +16 ms vs YARP) Cool A/B + userspace delay shim analysis Fast-path WriteResponse then CopyBody emitted a header-only TLS record before body; shim pays delayMs per read → ~3 extra 5 ms trips Materialize known-CL ≤64 KiB on fast path + coalesce headers+body (bc768069). Cool ≈ 1.00×; CI 32620889168: Win 663/664, Linux 1199/1196.
GHA compare-post/compare-arch failed; laptop H3 POST/slow passed Failed run logs (32602145518, 32602146550) (1) Dual-listen: TCP ephemeral then QUIC UDP same port → Windows WSAEADDRINUSE when UDP busy/excluded. (2) Incomplete StreamBodyWriter + DataAvailable==0 pooled origin sockets with unread CL → next request H3_INTERNAL_ERROR (HeadersRead slow-consumer + warmup cancel amplifies on 4 vCPU) Retry ephemeral TCP+QUIC bind in ProxyServer.Start/AddEndPoint; always close origin on incomplete StreamBodyWriter; YARP/nginx dual-stack free-port pick
H3→H1 latency bundle (skip drain / skip Flush / HEADERS+DATA coalesce) Cool absolute win (cool-h3h1-latency-bundle-20260823/) + CI bridges 32652931261 @ 3f948409 Cool c=64 TWP ~25k (tip ~20–23k); laptop YARP ~30k → cool TY ~0.85×. Trace @ c=64: p50 gap not exclusive CPU. CI miss: Win H3→H1 0.92× → ~0.87× (13,391 / 15,444). Lin H3→H1 still leads (~1.11×). Lossy Lin H3 improved 0.76×→0.89× (278/314) but still <1.00×. Reverted (2bf18d75).
H2→H1 Memory ~5–9× YARP at RPS parity Saturation RSS sampler + bag lifetime analysis ConcurrentBag<Task> PendingSynthetics/Finalizations retained completed Tasks (session closures) for client H2 conn life; full SessionEventArgs per stream on IsFastPath Http2PendingWork remove-on-complete; H2→H1 warm TryRentPooled + HeaderBuilder wire (H3→H1 analogue). Remeasure Block B Memory. Do not shrink windows / single-conn.
H3→H1 Win ~0.993× residual after one-pass QPACK Cool A/B + gen0 on post-status path Per-response MemoryStream QPACK builder + Latin-1 string round-trip + new byte[] tiny body ThreadStatic ResponseBlockBuilder rent; span AddHeader; BufferPool body when Available covers CL. Bridges CI @ 0ff3673c / 32685354747: Win 1.12× (27,046 / 24,085); Linux 1.09×.
H3→H3 Win CI ~0.85× YARP; MITM Full÷Reverse ≪0.80 on H2/H3/H1 arms (32960766249) Cool paired A/B (h3h3-dig/) + compare-product MITM arms H3→H3: per-frame QUIC I/O. MITM Full: probe-only fast path coupled product code to RPS harness header name; H2 !wouldInjectVia bail; H3 Via gate blocked preencoded relay with default pseudonym Landed v1 @ df172718: MitmCompressedRelayHelper append-only (max 4 unique headers); H2 zero-copy append suffix; all MITM arms ≥ 0.70× on GHA median (33041445371 ×3). Landed v2 @ acfb27e1 (#981): MitmStaticRebuildHelper drop-only static rebuild (max 4 unique drops) + non-unique trailing append (second Set-Cookie); modify value / body still full re-encode. Three MITM tiers: Lite (unchanged relay), Append/Strip-lite (append or drop-only on static GET), Full session (modify/body/multi-edit). v2 GHA medians @ acfb27e1 — Win 33087085235/88466/91622, Lin 88466/91622/33105885748; all MITM arms ≥ 0.70×; one Lin arm (H1 TLS→H2 plain Full÷Reverse) −6% vs v1 (0.931 vs 0.991), within harness noise.
H3→H1 sticky ConcurrentBag TCP pool (bypass factory poolLock) Cool tip-vs-sticky both orders @ c=32 (cool-sticky-vs-tip-20260823/) Hypothesis: CI 4 vCPU multiplex tax on poolLock/queue Cool flat (~1.07× both). Reverted.
H3→H1 Request/Response ConcurrentBag + QPACK ArrayPool EncodeResponse Cool same-thermal tip-vs-pool @ c=32 (cool-pool-qpack-20260823/) Hypothesis: HeaderCollection graph + QPACK byte[] gen0 under 32 wakeups Cool flat (tip ~1.08×, pool ~1.06×). TWP absolute sometimes +high single digits; ratio not ≥+2%. Reverted.
H3→H1 FIN-drain overlap with origin TCP + message pool Cool both orders (cool-pool-overlap-20260823/) Hypothesis: hide Quic FIN read behind origin FillBuffer (do not skip drain) Cool miss (order-noisy mean ≪ tip). Reverted.
H3→H1 one-pass H1 headers → QPACK (no Response/HeaderCollection) Cool tip-vs-onepass YT @ c=32 (cool-onepass-v2-20260823/) + H3 integ Post-status HeaderCollection graph + EncodeResponse under 32 wakeups (CI 4 vCPU gen0) Cool YT win: one-pass TWP÷YARP ≈ 1.24× (20,893/16,904) vs tip ≈ 1.18× (19,807/16,843); TWP absolute +~5%. TY polluted by soft YARP. Unit+H3 integ green.

Guardrails while optimizing

  • Real proxy improvements only — do not game the RPS harness. YARP (and nginx) are the yardstick, not the product. Land changes that reduce real work on the hot path for general reverse-proxy / keep-alive traffic: fewer allocations, fewer syscalls, less protocol waste, less interception tax when unused. Do not land knobs, special cases, or architecture copies whose only purpose is to inflate TWP÷YARP on the tiny-GET probe. Use the probe to find where time burns; revert experiments that do not help the proxy itself even if a noisy pair looks green.
  • Full unit + integration suites after every change. Several perf changes introduced real regressions (the scheme mismatch, the DATA race, HTTP-version and Content-Length bugs on the bridges); the suites and the benchmark's own error SLO caught all of them the same day.
  • A standalone external repro (tools/H2ExternalRepro) validates against real internet sites, which surface flow-control and settings behavior loopback benchmarks never exercise.
  • Wiki numbers carry their run IDs and an explanation of why each number moved, so a future regression has a baseline with provenance.

Checklist

  1. Re-baseline with paired same-thermal A/B before believing any gap.
  2. Sweep concurrency — let the curve's shape choose the tool (serialization → dumps; per-request cost → CPU sampling).
  3. dumpasync for where requests wait; dotnet-trace for where cycles burn.
  4. Decompose internal vs client-observed latency (TWP_RPS_STAGE_TIMING); a large gap means queueing upstream of the pipeline.
  5. Read the faster system's source to answer named hypotheses; keep TWP's architecture.
  6. Before keeping a change: confirm it is a real proxy improvement (less work / alloc / I/O on a general hot path), not a probe-only tweak to beat YARP. For Memory lites: keep only if RSS improves (or not worse) and RPS ÷YARP does not regress.
  7. For Memory: use harness proxy_rss_peak_bytes / descendant-tree sampler; confirm retention with gcdump when bags/sessions look sticky; do not shrink H2 windows or force single-connection to game RSS.
  8. Run the full test suites and the external repro before publishing; record run IDs in the wiki.
Add a custom footer
Add a custom sidebar

Clone this wiki locally