Performance Profiling

Performance Profiling

How the throughput hotspots behind the numbers on the Performance page were found. This page documents the techniques and tools so future performance work (or a regression hunt) can follow the same playbook instead of rediscovering it. Everything here was used in the 2026 pass that took the HTTP/2 bridge arms from ~6× behind the managed reverse peer to parity-or-close.

Contents

• The measurement harness
• Controlling measurement noise
• Local Windows lab (developer laptop)
• Architecture-sensitive
• Technique 1: concurrency sweep as a shape test
• Memory (RSS) — H2→H1 vs H1 / H3
• Technique 2: async dumps — find where requests wait
• Technique 3: per-stage latency decomposition
• Technique 4: CPU sampling
• Technique 5: reference-source comparison
• TWP vs YARP IO model
• Case studies: symptom → tool → root cause → fix
• Guardrails while optimizing
• Checklist

The measurement harness

All throughput work starts from RpsLoadProbe:

• Each arm is one topology: client protocol × origin protocol × TLS/cleartext × reverse/MITM (e.g. twp-reverse-http2-cleartext = H2 TLS client → H2→H1 bridge → cleartext H1 origin).
• Every TWP arm has a control arm — the managed reverse peer (and the native reverse peer where it can run the path) hosting the identical workload in the same session, so both sides see the same machine state.
• Every --ramp arm is three OS processes (parent load generator + origin child + proxy child) with a parent-seeded loopback CA (TWP_RPS_CERT_DIR). Combined --serve is debug-only. Absolute RPS from older combined TLS/QUIC-origin cells is not comparable to split runs — prefer TWP÷peer ratios.
• The probe ramps concurrency (typically c=8→64) and reports sustainable RPS: the last concurrency step that still met the error-rate and p99-latency SLO. A ramp that grows RPS but blows p99 is a queue, not throughput.
• Results land in timestamped CSVs under tools/RpsLoadProbe/results/. Publishable Performance tables cite GitHub Actions run IDs (median of 3 on matched 4 vCPU / 16 GiB runners). This page keeps the local Windows lab so a cool paired win can be proven before spending a CI remasure.

# one suite
pwsh tools/RpsLoadProbe/run-rps.ps1 -Mode compare-bridges
# one arm, custom ramp (apphost, not `dotnet <dll>` — the child processes re-exec the host)
tools/RpsLoadProbe/bin/Release/net10.0/RpsLoadProbe.exe --ramp --mode reverse-http2-cleartext `
  --concurrency 8,16,32,64 --warmup-sec 2 --duration-sec 5 --results-dir tools/RpsLoadProbe/results

Controlling measurement noise

On a laptop, thermal throttling dominates everything else: the same arm measured 23k, 11k, and 51k RPS in one afternoon depending on accumulated heat. Rules that kept conclusions honest:

• Compare only within one back-to-back session. Never compare a number from this run against a number from an hour ago.
• Prefer TWP÷peer ratios over absolutes. The control arm soaks up the same throttling.
• For a targeted A/B question, run the two arms paired: cooldown (~2 min idle), arm A, arm B immediately after — and alternate which goes first across repeats so heat bias cancels. This is how "MITM costs 0.65–0.75× of its reverse twin, and the delta is purely the extra origin TLS leg" was established: the two probe arms differ by exactly one flag (ForwardCleartext).
• If an arm's ratio looks newly bad, re-measure before profiling — several "regressions" were heat.
• Gate before publishing: optimize against cool paired ratios on the laptop below. After a cool win, remasure on matched Windows+Linux GHA (workflow_dispatch RPS saturation) and paste medians into Performance. Windows GHA removes laptop thermal skew; it is not the same as a cool pair — shared-VM noise still applies, so prefer TWP÷peer ratios.

Local Windows lab (developer laptop)

Local debug setup and historical High-perf / cool-paired tables. Do not paste these absolutes onto Performance — that page is CI-only. Use this section to iterate: cool A/B, then publish from GHA.

Measurement environment

OS	Windows 11 (10.0.26200)
CPU	11th Gen Intel Core i7-1185G7 @ 3.00 GHz (8 logical processors)
RAM	31.8 GiB
Runtime	.NET 10.0.10
nginx	nginx/Windows 1.31.3
YARP	Yarp.ReverseProxy 2.3.0
Harness	RpsLoadProbe Release; arms run sequentially

This box is 8 logical / ~32 GiB — not the 4 vCPU / 16 GiB GHA class. Treat ratios as the local gate; remasure on CI before claiming a publishable win.

Cool = ~2 min idle, then paired A/B (alternate who goes first; mean of both orders @ c=32) — authoritative local gate; reverse tiny-GET / body cells below use those cool absolutes when cited. Heated = long sequential matrix (thermal skew). 🥇 = higher cool sustain in that row (or heated sustain only when no cool pair exists — noted).

Windows — Titanium vs nginx vs YARP (laptop)

Client / origin: HTTP version and whether TLS is used (plain = cleartext, TLS = encrypted, QUIC = HTTP/3).

Reverse TWP/YARP cells for cool-audited arms are cool paired means (see notes). nginx cells remain the older heated High-perf baseline (nginx was not in the cool pairs). MITM rows are heated 1-rep full matrix (windows-20260822-mitm-full/, probe @ 1b5ca9f9) — same 15 Client×Origin pairs as Reverse plus dual-crypto extras.

Mode	Client	Origin	TWP sustain	TWP peak	nginx sustain	nginx peak	YARP sustain	YARP peak
Reverse	HTTP/1 · plain	HTTP/1 · plain	🥇 41,390	41,390	15,196	18,806	38,772	38,772
Reverse	HTTP/1 · plain	HTTP/1 · TLS	🥇 30,844	30,844	Not possible	Not possible	30,621	30,621
Reverse	HTTP/1 · TLS	HTTP/1 · plain	🥇 35,205	35,205	10,252	13,741	29,750	29,750
Reverse	HTTP/1 · TLS	HTTP/2 · TLS	🥇 25,540	25,540	Not possible	Not possible	24,920	24,920
Reverse	HTTP/1 · TLS	HTTP/3 · QUIC	🥇 21,819	21,819	Not possible (no QUIC)	Not possible	20,712	20,712
Reverse	HTTP/2 · plain	HTTP/1 · plain	🥇 46,517	46,517	Not possible	Not possible	42,994	42,994
Reverse	HTTP/2 · plain	HTTP/2 · plain	🥇 100,568	100,568	Not possible	Not possible	86,021	86,021
Reverse	HTTP/2 · plain	HTTP/2 · TLS	🥇 88,006	88,006	Not possible	Not possible	84,634	84,634
Reverse	HTTP/2 · plain	HTTP/3 · QUIC	🥇 27,493	27,493	Not possible (no QUIC)	Not possible	24,535	24,535
Reverse	HTTP/2 · TLS	HTTP/1 · plain	🥇 49,548	49,548	15,793	15,793	49,072	49,072
Reverse	HTTP/2 · TLS	HTTP/2 · plain	🥇 94,238	94,238	Not possible	Not possible	81,266	81,266
Reverse	HTTP/2 · TLS	HTTP/3 · QUIC	🥇 30,813	30,813	Not possible (no QUIC)	Not possible	24,039	24,039
Reverse	HTTP/3 · QUIC	HTTP/1 · plain	22,325	22,325	Not possible (no QUIC)	Not possible	🥇 23,773	23,773
Reverse	HTTP/3 · QUIC	HTTP/2 · TLS	🥇 22,297	22,297	Not possible (no QUIC)	Not possible	21,914	21,914
Reverse	HTTP/3 · QUIC	HTTP/3 · QUIC	🥇 26,299	26,299	Not possible (no QUIC)	Not possible	14,942	14,942
MITM	HTTP/1 · plain	HTTP/1 · plain	33,484	33,484	Not possible (no MITM)	Not possible	Not possible (no MITM)	Not possible
MITM	HTTP/1 · plain	HTTP/1 · TLS	38,193	38,193	Not possible (no MITM)	Not possible	Not possible (no MITM)	Not possible
MITM	HTTP/1 · TLS	HTTP/1 · plain	36,037	36,037	Not possible (no MITM)	Not possible	Not possible (no MITM)	Not possible
MITM	HTTP/1 · TLS	HTTP/2 · TLS	42,652	42,652	Not possible (no MITM)	Not possible	Not possible (no MITM)	Not possible
MITM	HTTP/1 · TLS	HTTP/3 · QUIC	28,114	28,114	Not possible (no QUIC)	Not possible	Not possible (no MITM)	Not possible
MITM	HTTP/2 · plain	HTTP/1 · plain	45,905	45,905	Not possible (no MITM)	Not possible	Not possible (no MITM)	Not possible
MITM	HTTP/2 · plain	HTTP/2 · plain	98,069	98,069	Not possible (no MITM)	Not possible	Not possible (no MITM)	Not possible
MITM	HTTP/2 · plain	HTTP/2 · TLS	89,055	89,055	Not possible (no MITM)	Not possible	Not possible (no MITM)	Not possible
MITM	HTTP/2 · plain	HTTP/3 · QUIC	41,662	41,662	Not possible (no QUIC)	Not possible	Not possible (no MITM)	Not possible
MITM	HTTP/2 · TLS	HTTP/1 · plain	51,042	51,042	Not possible (no MITM)	Not possible	Not possible (no MITM)	Not possible
MITM	HTTP/2 · TLS	HTTP/2 · plain	97,015	97,015	Not possible (no MITM)	Not possible	Not possible (no MITM)	Not possible
MITM	HTTP/2 · TLS	HTTP/3 · QUIC	40,358	40,358	Not possible (no QUIC)	Not possible	Not possible (no MITM)	Not possible
MITM	HTTP/3 · QUIC	HTTP/1 · plain	22,792	22,792	Not possible (no QUIC)	Not possible	Not possible (no MITM)	Not possible
MITM	HTTP/3 · QUIC	HTTP/2 · TLS	34,225	34,225	Not possible (no QUIC)	Not possible	Not possible (no MITM)	Not possible
MITM	HTTP/3 · QUIC	HTTP/3 · QUIC	21,164	21,164	Not possible (no QUIC)	Not possible	Not possible (no MITM)	Not possible
MITM	HTTP/1 · plain (CONNECT)	HTTP/1 · TLS	27,238	27,238	Not possible (no MITM)	Not possible	Not possible (no MITM)	Not possible
MITM	HTTP/1 · TLS	HTTP/1 · TLS	29,186	29,186	Not possible (no MITM)	Not possible	Not possible (no MITM)	Not possible
MITM	HTTP/2 · TLS	HTTP/2 · TLS	79,070	79,070	Not possible (no MITM)	Not possible	Not possible (no MITM)	Not possible
MITM	HTTP/2 · TLS	HTTP/1 · TLS	38,943	38,943	Not possible (no MITM)	Not possible	Not possible (no MITM)	Not possible
MITM	HTTP/3 · QUIC	HTTP/1 · TLS	19,571	19,571	Not possible (no QUIC)	Not possible	Not possible (no MITM)	Not possible

Windows reverse tiny-GET: base matrix 2026-08-20 High-perf, Linux-matched harness (warmup 2s / measure 8s; concurrency 8, 16, 32, 64; median of 3 repeats except H2 TLS→H3 and H3→H1/H2, which have 2). CSVs under tools/RpsLoadProbe/results/windows-20260820/ (compare-same, compare-bridges). MITM and heavier reverse: 1-repeat follow-up under windows-20260820-quick/. Absolute RPS swings with sequential-arm heat; prefer TWP÷YARP ratios.

2026-08-21 remasure (through exact-body + H3 QPACK-normalized names): H1 plain, H1 TLS, H1→H2, H3→H1, H3→H2 refreshed as mean of both arm orders at c=32 (win-final-*). Exact-size H2 origin body materialize (no MemoryStream+ToArray) and HeaderNamesAreHttp2Normalized on the H3 fast Request. Other reverse Windows rows still 2026-08-20 unless noted.

2026-08-22 matrix fill (missing plain cells): Library fix so cleartext-listen reverse (DecryptSsl=false) honors ForwardCleartext=false as origin HTTPS (H1 plain→HTTPS). New probe arms: reverse-http1-to-https / yarp-reverse-http1-to-https, http-mitm (explicit plain→plain). Full Windows compare-same + compare-bridges + plain twins under tools/RpsLoadProbe/results/windows-20260822-matrix/ (1-rep; warmup 2s / measure 8s; c=8,16,32,64).

2026-08-22 MITM full matrix: CompareMitm matches Reverse’s 15 Client×Origin pairs (inspectable/decrypt) plus CONNECT / TLS↔TLS extras. Laptop 1-rep CSV: tools/RpsLoadProbe/results/windows-20260822-mitm-full/. Publishable CI medians (Win+Linux, 3-rep): 32588707712 @ 1b5ca9f9 — see Performance.

Load generators: Reverse inbound H3 arms use **dotnet-httpclient** (http_version=3.0, RequestVersionExact) after dual-listen reverse H3. MITM H3→H2 / H3→H3 / H3→H1 plain reuse the same dual-listen transparent reverse path as their reverse twins (ForwardCleartext / decrypt knobs). Older UDP-only quic-http3 MITM H3→H1 TLS numbers are dual-crypto extras (mitm-http3-to-http1).

Matched HttpClient TWP÷YARP — table cells are cool absolutes where cited: parity audit win-parity-audit-20260822-004214/ (both orders @ c=32): H1 plain 41,390 / 38,772 ≈ 1.07×; H1 TLS 35,205 / 29,750 ≈ 1.18×; H1→H3 21,819 / 20,712 ≈ 1.05×; H3→H3 26,299 / 14,942 ≈ 1.76× (YARP soft — treat absolute cautiously). 2026-08-22 cool paste win-cool-paste-20260822-063226/ (both orders @ c=32): H1→H2 25,540 / 24,920 ≈ 1.02×; H1 plain→HTTPS 30,844 / 30,621 ≈ 1.01×; h2c→H3 27,493 / 24,535 ≈ 1.12×; H2 TLS→H3 30,813 / 24,039 ≈ 1.28×; h2c→H1 46,517 / 42,994 ≈ 1.08×; H3→H1 22,325 / 23,773 ≈ 0.94×; H3→H2 22,297 / 21,914 ≈ 1.02×. 2026-08-23 soft coolish (both orders @ c=32, after session-lite H2/H3 gate): H3→H1 ≈ 1.09×; h2c→H1 ≈ 1.05×; H1→H3 ≈ 1.25×. Published CI Win bridges @ 11e32f1c still show those three ≤1.00× — tip remasure @ 62e5efcd in flight. TWP-led H2 same-protocol rows unchanged (h2c↔h2c ≈ 1.17×, etc.).

Attempted H1→H3 micro-opts (2026-08-22, reverted): Lowercasing H1 request names before QPACK + buffering tiny H3 origin bodies without draining to FIN regressed cool H1→H3 from ~1.13× to ~0.65× — kept out. 2026-08-23 kept: same ≤64 KiB eager materialize in ForwardOverQuicAsync plus drain-to-FIN before Dispose (else RST poisons the QUIC pool → handshake-per-request; first attempt 1.16×→0.7×). Cool both orders ≈ 1.03–1.23× (cool-h3-origin-eager64-drain-20260823/). Smoke H2→H3 / H3→H3 still lead; H3→H1 unchanged (~0.98× TY).

2026-08-22 H3 bridge hot-path (kept): Decode H2 origin HEADERS into the Response HeaderCollection (no second collection + copy). H3→H1 fast path: drain chunked/connection-close origin bodies before pool Release (Kestrel WriteAsync often chunked — empty DATA was a correctness bug); lowercase H1 response names once for QPACK; skip GetOriginHostPort on warm pool hit. Cool CSVs: tools/RpsLoadProbe/results/win-h3h1-postfix-20260821-231146/, win-h3h1-yarpfirst-20260821-231420/.

MITM÷TWP reverse twin (2026-08-22 full matrix): Prefer same-session ratios. Heated mitm-full absolutes: plain→plain (http-mitm) 33,484; CONNECT https-mitm 27,238; H1 TLS↔TLS 29,186; H2 TLS↔TLS 79,070; H2→H1 TLS 38,943; H3→H1 TLS 19,571. Matched Client×Origin MITM cells sit near their reverse twins when topology is inspectable-cleartext (e.g. h2c↔h2c MITM 98,069 vs reverse cool 100,568).

Why H3 absolute RPS ≪ H2 on this box: tiny-GET loopback is not where H3 wins. Cool paired H3→H3 now leads YARP (~1.76× on the soft 2026-08-22 audit; High-perf matrix still shows ~0.95×). MsQuic + dual QUIC hops dominate absolute RPS vs H2 same-protocol (~90–100k), not TWP architecture.

nginx/Windows is a limited port — use it for same-OS comparison only, not as the industry nginx baseline.

H2 TLS → H1 plain on Windows: fair terminate — TWP leads sustain (~1.01× YARP) in the current table. Absolute RPS swings with background load; treat as same-OS only.

Heavier reverse GET (64 KiB / 256 KiB)

1-repeat; warmup 2s / measure 8s; concurrency 8–64. Source: windows-20260820-quick/compare-bodies.

Body	Client	Origin	TWP sustain	TWP peak	nginx sustain	nginx peak	YARP sustain	YARP peak
64 KiB	HTTP/1 · TLS	HTTP/1 · plain	11,414	12,184	1,119	1,184	🥇 12,664	13,403
64 KiB	HTTP/2 · TLS	HTTP/1 · plain	🥇 7,744	7,744	1,030	1,063	6,844	6,844
64 KiB	HTTP/3 · QUIC	HTTP/1 · plain	1,109	1,109	Not possible (no QUIC)	Not possible	🥇 3,108	3,108
256 KiB	HTTP/1 · TLS	HTTP/1 · plain	3,260	3,260	292	301	🥇 3,882	3,934
256 KiB	HTTP/2 · TLS	HTTP/1 · plain	1,935	1,935	236	239	🥇 2,179	2,179
256 KiB	HTTP/3 · QUIC	HTTP/1 · plain	771	771	Not possible (no QUIC)	Not possible	🥇 939	939

nginx/Windows collapses on large reverse bodies in this harness; treat as same-OS only.

2026-08-22 cool remasure (bodies) — H2→H1 cells are cool means (win-bodies-coalesce288-20260822/, both orders @ c=32): 64 KiB 7,744 / 6,844 ≈ 1.13× → TWP leads; 256 KiB 1,935 / 2,179 ≈ 0.89× → YARP leads. H1 TLS→H1 64 KiB still heated (marker follows heated). H3→H1 64 KiB cool ≈ 0.96× → YARP leads. 256 KiB H1/H3: heated → YARP leads.

POST 64 KiB request + 64 KiB response

1-repeat; warmup 2s / measure 8s. Source: windows-20260820-quick/compare-post.

Client	Origin	TWP sustain	TWP peak	nginx sustain	nginx peak	YARP sustain	YARP peak
HTTP/1 · TLS	HTTP/1 · plain	🥇 6,003	6,160	383	413	5,264	5,420
HTTP/2 · TLS	HTTP/1 · plain	3,519	3,519	357	389	🥇 4,741	4,871
HTTP/3 · QUIC	HTTP/1 · plain	🥇 1,973	2,001	Not possible	Not possible	1,802	1,893

H1 POST: TWP leads (heated and cool). H2 POST: YARP leads — heated ≈ 0.74×, cool ≈ ~0.88–0.95× with c=1 TWP ahead (~1.2×); residual is multiplex scaling, not single-stream cost. H3 POST (2026-08-22): UpdateContentLength on streamed uploads stamped CL=0 (ab16a871). Heated remasure sustain0-verify/h3-post/ (c=8–64): TWP sustain 1,973 / YARP 1,802 ≈ 1.09×.

Lossy / high-RTT (H2 HOL / H3 packet loss)

Userspace 5 ms one-way delay + 1% stall (TCP) or datagram drop (UDP/QUIC); 64 KiB GET. 1-repeat; warmup 2s / measure 8s; c=8–64. Source: windows-20260822-lossy-h3-quic/ (lossy H3 forced to quic-http3).

Client	Origin	TWP sustain	TWP peak	nginx sustain	nginx peak	YARP sustain	YARP peak
HTTP/1 · TLS	HTTP/1 · plain	578	578	500	500	🥇 656	656
HTTP/2 · TLS	HTTP/1 · plain	14	15	14	14	🥇 15	15
HTTP/3 · QUIC	HTTP/1 · plain	🥇 1,572	1,572	Not possible (no QUIC)	Not possible	0	50

H1 stays usable; H2 collapses under connection stalls (HOL). H3 is the protocol-shape win: TWP H3 sustain ≈ 112× H2 on the same lossy session (datagram drop, not HOL). YARP H3 did not hold the p99 SLO under this userspace UDP shim (peak 50) — treat as same-session measurement, not a capability claim. Absolute RPS is low because the shim delays every buffer/datagram.

Architecture-sensitive

compare-arch (1-repeat; warmup 2s / measure 8s; c=8,16,32,64). Source: windows-20260822-arch/. Slow consumer = 256 KiB GET, client reads 16 KiB then sleeps 8 ms. Early response = 64 KiB POST, origin writes after the first 8 KiB. Duplex H2 = overlapping 64 KiB POST on H2 TLS↔H2 TLS. WebSocket = echo round-trips/sec on H1 TLS→H1 plain /ws. See TWP vs YARP IO model.

Scenario	Client	Origin	TWP sustain	TWP peak	nginx sustain	nginx peak	YARP sustain	YARP peak
Slow consumer (256 KiB GET, throttled client read)	HTTP/1 · TLS	HTTP/1 · plain	175	175	🥇 203	203	196	196
Slow consumer (256 KiB GET, throttled client read)	HTTP/2 · TLS	HTTP/1 · plain	🥇 248	248	213	213	🥇 248	248
Slow consumer (256 KiB GET, throttled client read)	HTTP/3 · QUIC	HTTP/1 · plain	🥇 248	248	Not possible (no QUIC)	Not possible	🥇 248	248
Early response (origin writes after first request chunk)	HTTP/1 · TLS	HTTP/1 · plain	🥇 6,402	6,606	270	270	5,135	5,135
Early response (origin writes after first request chunk)	HTTP/2 · TLS	HTTP/1 · plain	2,938	3,330	117	141	🥇 4,056	4,056
Early response (origin writes after first request chunk)	HTTP/3 · QUIC	HTTP/1 · plain	🥇 1,794	1,794	Not possible (no QUIC)	Not possible	1,485	1,485
Duplex (both directions live)	HTTP/2 · TLS	HTTP/2 · TLS	9	590	Not possible	Not possible	🥇 2,455	2,455
Duplex (WebSocket / extended CONNECT)	HTTP/1 · TLS	HTTP/1 · plain	🥇 38,235	38,823	18,251	19,054	37,803	38,454

Slow consumer is sleep-bound (~16 × 8 ms per 256 KiB); H1/H2 sit in the same band. H3 slow-consumer (2026-08-22): fast path closed the origin socket for CL>16 KiB without StreamBodyWriter (36d21f67); remasure sustain0-verify/h3-slow/ matches YARP at 248 sustain. Early-response H1: TWP leads (~1.25× YARP) — sequential H1 still finishes the exchange quickly when the origin answers after 8 KiB. H3 early-response (2026-08-22): cool mean ≈ 1.21× YARP after overlapping origin upload with ReceiveResponse / StreamBodyWriter (fix-early-tls/). Early-response H2 and duplex H2: YARP leads on heated matrix; TWP H2↔H2 duplex sustain 9 vs peak 590 (errors at higher concurrency) vs YARP 2,455. WebSocket echo: TWP leads (~1.01× YARP); nginx/Windows same-OS only.

TLS termination cost (H1 TLS → cleartext origin)

1-repeat. Source: windows-20260820-quick/compare-tls-cost.

Workload	TWP sustain	TWP peak	nginx sustain	nginx peak	YARP sustain	YARP peak
Keep-alive · tiny GET	35,724	39,386	17,148	18,318	🥇 39,905	39,905
New-connection · tiny GET	926	926	789	800	🥇 1,088	1,088
Keep-alive · 256 KiB GET	3,349	3,349	291	299	🥇 3,916	3,916

Technique 1: concurrency sweep as a shape test

Cheapest tool with the highest information density. Run the arm at c=1 and at c=64 and look at the shape, before reaching for any profiler:

Observation	Meaning
Slow at c=1 and c=64 by the same factor	Per-request cost (allocations, crypto, syscalls) — go CPU-profile it
Faster at c=1 but flatlines while the control arm scales	A serialization point — something processes streams one at a time; profilers of per-request cost will mislead you
Scales to a cliff, then errors/SLO failures	Resource exhaustion or a convoy (locks, pool limits, flow-control windows)

The h2c→H1 bridge showed the second shape: TWP beat the managed reverse peer at c=1 (6,425 vs 5,449 RPS) but flatlined at ~22k while the managed reverse peer scaled to 46k. That single observation eliminated allocation work, System.IO.Pipelines, and syscall efficiency as hypotheses and said "find the serial section."

Memory (RSS) — H2→H1 vs H1 / H3

Technique: connection-scoped Task retention + H2 bridge lite wire

Tools already on this page (cool A/B, concurrency sweep, dumpasync, stage timing) find where time burns. Memory gaps need the harness RSS sampler plus heap confirmation:

# Saturation / matrix CSV columns (every measure step):
#   proxy_rss_peak_bytes, proxy_cpu_avg_pct
pwsh tools/RpsLoadProbe/run-rps.ps1 -Mode compare-saturation

# Optional heap confirm under H2→H1 load (Full dump):
dotnet-gcdump collect -p <proxy PID>
# or: dotnet-dump collect -p <proxy PID> --type Full → dumpheap -stat

Root causes (H2→H1 ≫ YARP Memory while RPS ≈ parity):

1. PendingSynthetics / PendingFinalizations retained completed Tasks for the life of each client H2 connection (ConcurrentBag never removed). Fixed with Http2PendingWork (remove-on-complete tracker) — connection-scoped retention, not a process-immortal leak.
2. Per-stream SessionEventArgs on H2→H1 / H2→H3 even when IsFastPath (H1 keep-alive uses ResetForKeepAlive; H3 inbound already had H3H2FastForward). Mitigated with warm TryRentPooled + HeaderBuilder wire on interception-off bodiless H2→H1 (H3→H1 analogue) and IsFastPath skips on H2→H3; full decode-time session skip still requires deeper Http2Helper work.
3. Custom H2 stack per client connection vs Kestrel pooled streams (residual).
4. Not the 768 KiB flow-control windows — advertised credit, not tiny-GET buffers. Do not force single H2 connection to game RSS (laptop A/B: −14% RSS / −23% RPS).

Keep / revert for new lites: cool-measure Memory + ÷YARP RPS; revert the lite if Memory is no better (within noise) or RPS regresses. Bag tracker is RPS-neutral retention — keep regardless.

Published Saturation control Memory (RSS) @ a3b9af1e / remasure (pre-fix): TWP H1 ≈ YARP or lower; TWP H2→H1 ~5–9× YARP; TWP H3→H1 ~1.1–1.7×. Remeasure after bag tracker + H2 lite wire and paste Block A/B/C below (laptop) and on Performance (CI).

Local saturation control (laptop)

Same shape as Performance § Saturation control; one OS = this laptop. Fill after cool compare-saturation (median of repeats; Memory/CPU at peak-RPS step).

pwsh tools/RpsLoadProbe/run-rps.ps1 -Mode compare-saturation

Block A — H1 plain

1-rep cool-ish laptop @ 0ff3673c (laptop-sat-memory-fix/, 2026-08-24). Absolutes are local-only; prefer ratios.

Arm	Generator	Sustain	Peak	% of origin-HttpClient	Memory (RSS)	CPU avg %
origin-direct	dotnet-httpclient	47,946	62,661	100%	58 MiB	32.2
bare-reverse-http1	dotnet-httpclient	36,177	36,177	57.7%	57 MiB	35.4
nginx-reverse-http1	dotnet-httpclient	18,208	19,436	31.0%	217 MiB	🥇 12.1
yarp-reverse-http1	dotnet-httpclient	29,592	34,703	55.4%	85 MiB	41.3
twp-reverse-http1	dotnet-httpclient	34,061	34,061	54.4%	🥇 81 MiB	39.4

Block B — H2 TLS→H1

Same run. TWP Memory ~199 MiB vs prior laptop multi-conn ~425 MiB / CI ~848 MiB — bag drain + H2→H1 lite wire; RPS still 1.04× YARP.

Arm	Generator	Sustain	Peak	÷YARP	÷nginx	Memory (RSS)	CPU avg %
nginx-reverse-http2	dotnet-httpclient	3,186	11,389	0.25×	1.00×	243 MiB	🥇 11.4
yarp-reverse-http2	dotnet-httpclient	45,269	45,269	1.00×	3.98×	🥇 105 MiB	48.3
twp-reverse-http2-cleartext	dotnet-httpclient	🥇 46,987	46,987	1.04×	4.13×	199 MiB	47.3

Block C — H3→H1

Same sequential run (thermal; treat ÷YARP cautiously vs cool pair).

Arm	Generator	Sustain	Peak	÷YARP	Memory (RSS)	CPU avg %
yarp-reverse-http3-cleartext	dotnet-httpclient	🥇 28,101	28,101	1.00×	🥇 172 MiB	45.4
twp-reverse-http3-cleartext	dotnet-httpclient	22,910	23,628	0.84×	198 MiB	🥇 40.7

Laptop cool-ish A/B before bag/lite fix (2026-08-23, c=64, 8 s) — superseded by Block B once filled:

Arm	Memory (RSS)	RPS
TWP H2→H1 (EnableMultipleHttp2Connections=true, default)	425 MiB	41k
TWP H2→H1 (TWP_RPS_SINGLE_HTTP2_CONNECTION=1)	367 MiB	32k
YARP H2→H1 (multi)	104 MiB	41k

Technique 2: async dumps — find where requests wait

CPU profilers show where cycles burn; under async I/O the bottleneck is usually where requests park. Capture the async state machine population under load:

# must be a Full dump; a mini dump lacks the heap metadata dumpasync needs
dotnet-dump collect -p <proxy PID> --type Full
dotnet-dump analyze <dump file>
> dumpasync --stats

Read the histogram of parked continuations. In the bridge investigation, hundreds of in-flight requests were parked in synthetic-response emission waiting on one SemaphoreSlim (the client write lock) — a classic convoy, with the side signature of high system CPU from many tiny socket writes. The fix (a dedicated per-direction frame writer draining a channel and coalescing up to 32 frames / 32 KB per socket write, Http2FrameWriter) was worth 3.4× on that arm.

Technique 3: per-stage latency decomposition

When internal work looks fast but clients still see high latency, decompose the request path. TWP already captures per-request milestones when EnableRequestTimingCapture is set (see Request timing); RpsLoadProbe has an opt-in collector that aggregates them under load:

# any non-empty value enables; a path (length > 1) writes reports to that file
$env:TWP_RPS_STAGE_TIMING = "C:\temp\stage-timing.log"

StageTimingCollector subscribes to AfterResponse, buckets HttpRequestTiming durations (client read, connection wait, request send, TTFB, delivery, total), and prints p50/p90/p99 per stage every 20 s. Subscribing to AfterResponse disables the no-interception fast path, so this must stay out of publishable runs.

The decisive read: the internal pipeline showed p50 87 µs per request while clients observed p50 2.6 ms — so ~2.5 ms of queueing happened before a stream entered the instrumented pipeline. That pointed at the per-connection HTTP/2 frame loop, which was running each stream's BeforeRequest handler prefix inline (~44 µs per HEADERS frame), capping any single client connection at ~22k streams/s regardless of concurrency. Starting the handler on the thread pool from the already-ordered dispatch task took the arm from 22k to 47k RPS.

The same collector separates "proxy is slow" from "origin leg is slow": on the H1→H2 bridge, TTFB was 240 µs at c=8 but 1,830 µs at c=64 with barely more RPS — the signature of CPU saturation, not another serial section.

Technique 4: CPU sampling

For arms where the sweep says "per-request cost" or "saturation," attach the sampler during a long window (the ramp's default 5–7 s steps are too short to attach; use a 150 s single-concurrency run):

tools/RpsLoadProbe/bin/Release/net10.0/RpsLoadProbe.exe --ramp --mode <arm> --concurrency 64 `
  --warmup-sec 2 --duration-sec 150 --results-dir tools/RpsLoadProbe/results/profiling
# ramp logs print: attach: split origin pid=… proxy pid=…
# in a second shell:
dotnet-dump collect -p <proxy PID> --type Full
dotnet-trace collect -p <proxy PID> --profile dotnet-sampled-thread-time --duration 00:00:25

This is a confirmation tool more than a discovery tool here: it confirmed the residual H1→H2 gap after origin-connection sharing is still whole-box cost (dual TLS legs plus the per-request session pipeline). Sharing lifted the arm from 0.33× to 0.53× peer at c=32 (rps-ramp-20260818-130040 / 130112); cool remeasure after grow-at-4 stayed ~0.51× (profile-baseline / profile-post-fix). TTFB still rises with concurrency. At c=32 dumpasync showed 8 origin ReadLoopAsync instances (pool already spreading) plus Monitor / SslStream in the sampled stacks — not a single-conn convoy. Honest remainder: dual-TLS + session cost on this 8-thread box.

Technique 5: reference-source comparison

When a comparable managed reverse peer is faster, read its source to answer named hypotheses — not to port its architecture. Two examples from this pass:

• "Does the reference .NET server stack tune MAX_CONCURRENT_STREAMS dynamically?" No — it opens additional origin connections when the stream limit is hit. TWP replicated the behavior within its own design (Http2OriginRelayPool).
• "Is System.IO.Pipelines the advantage?" No — TWP's buffered HttpStream already amortizes socket reads to one syscall per buffer drain; the memcpy ReadOnlySequence would remove costs ~0.02% of a request, and the TLS decrypt copy exists in both models (SslStream cannot produce a ReadOnlySequence; the reference .NET server stack copies decrypted bytes into its Pipe too). Measured support: H1 arms at parity, and TWP's c=1 latency lower than the managed reverse peer's. The layering difference is real; it is not why the tiny-GET tables look the way they do — see TWP vs YARP IO model.
• "When does the managed reverse peer open another origin H2 connection?" ForwarderHttpClientFactory sets EnableMultipleHttp2Connections = true by default — SocketsHttpHandler grows sessions under stream pressure. TWP's PoolGrowActiveStreamThreshold is the analogous dial (lowered 16→4 after profiling).

TWP vs YARP IO model

Architecture context for future gap hunts — not a root-cause claim. YARP sits between Kestrel (System.IO.Pipelines on the inbound connection) and SocketsHttpHandler, with HttpForwarder / StreamCopier able to run request and response pumps at once. TWP's classic HTTP/1 session is one async state machine: send request, copy request body, wait for response, copy response body (HandleHttpSessionRequest / CopyBodyAsync). That is leaner for half-duplex HTTP, which is most of the RPS probe.

Do not over-count the tax:

• Kestrel's transport read/write loops are mostly per connection, not four new tasks per tiny GET. Concurrent copiers matter when there is a body or true duplex.
• Extra Tasks are extra continuations, not extra OS threads. Both stacks yield on I/O.
• TWP H2/H3 is already multi-task (frame intake, Http2FrameWriter, per-stream dispatch, origin read loops). WebSockets already run two CopyToAsync relays (TcpHelper.SendRawTap). The "one worker" story is H1 request/response, not the whole product.
• A pipe's real win is a bounded buffer between stages (writer pauses when the reader is slow). await WriteAsync in a copy loop already stalls the next read. TWP already has BoundedBodyPipe and bounded H2→H1 DATA channels (FullMode = Wait).

YARP inherits better insurance for slow consumers, protocol-edge bridging, and full-duplex (gRPC / streaming POST / response-starts-early) because those behaviors come with Kestrel + the forwarder. That is a workload preference, not a ceiling. Match it locally without becoming YARP:

Workload	Do	Do not
Tiny-GET / H1 POST	Keep the sequential session. It already wins or ties.	Put a pipe + two copiers under every request.
Slow consumer / lossy H1	Bound the origin→client copy; keep frame-writer coalesce.	Blame "missing Kestrel" before measuring buffer/backpressure.
Protocol bridges	Keep the direct bridges; fix unfinished paths (e.g. H3 POST sustain 0).	Port Kestrel for speed — several bridges already lead YARP.
True duplex	Start both pumps while both directions are live (same pattern as WebSockets).	Force one linear task to own a bidirectional stream.

Remaining YARP-led cells in this pass (H2 POST at c=32, some 256 KiB bodies, lossy H1) were named as multiplex/shared-writer, copy/syscall/coalesce, or buffer-vs-delay — not "we lack pipelines." nginx still leads both on Linux H1; if there is a hard ceiling it is managed C# vs native, not TWP vs pipes.

The architecture-sensitive table (and the CI medians on Performance) is compare-arch. A YARP win on duplex H2 is the concurrent-copier case; a TWP win on tiny-GET or WebSocket echo is not proof the sequential H1 session covers overlap.

Case studies: symptom → tool → root cause → fix

Symptom	Tool that found it	Root cause	Fix
h2c→H1 bridge 8.5k vs peer 46k, high system CPU	dotnet-dump + dumpasync --stats	Response emission convoy on the client write lock; many tiny socket writes	Queue all response frames through Http2FrameWriter (coalesced writes) — 3.4×
Same arm flat at ~22k at every concurrency, but faster than peer at c=1	Concurrency sweep + stage timing (87 µs internal vs 2.6 ms observed)	Frame loop ran each stream's BeforeRequest prefix inline (~44 µs/HEADERS)	Start the handler on the thread pool from the ordered dispatch task — 22k → 47k
External-site H2 downloads stalled at exactly 64 KB	Standalone repro tool (tools/H2ExternalRepro) + a window-size env knob	Flow-control starvation: batched WINDOW_UPDATE threshold larger than the default 65,535 B window	Advertise a reference .NET server stack-class 768 KiB initial stream window in both directions
Two bridge arms at 100% errors after the passthrough change	The benchmark suite itself (error-rate SLO)	:scheme mismatch in compressed header relay on mixed-transport bridges	Detect and re-encode the header block with the corrected scheme
POST arm collapsed 842 → 9 RPS	Benchmark suite + targeted repro	Client DATA frames raced the handler dispatch and were routed before channels existed	Await the stream's dispatch task before routing its DATA frames
H1→H2 bridge stuck at ~0.3× peer	Stage timing (TTFB 240 µs → 1,830 µs as c grows) + dotnet-trace	Dual TLS + per-request pipeline; also one dedicated origin H2 connection per H1 client	Shared Http2OriginConnectionPool (0.33× → 0.53× at c=32). Remainder still looks CPU-bound
H3→H2 SLO-failed above c=16 (then 100% errors after pooling)	Error log (TWP_H3_ERROR_LOG) + RFC 7540 §5.1.1	Exclusive ConcurrentBag cap 16, then concurrent SendAsync allocated stream ids off the write lock so a higher id's HEADERS could hit the wire first; the reference .NET server stack implicitly closed the lower idle streams and GOAWAYed	Shared pool (no exclusive checkout) + allocate stream id and write opening HEADERS under the same write lock. Reverse H3→H2 holds c=64 at 0% errors (rps-ramp-20260818-130231)
Inbound H3 ~0.40× peer blamed on “managed QUIC vs MsQuic”	Code read of QuicClientHandler.ListenQuic	Inbound H3 already is System.Net.Quic / MsQuic. Pre-match ratios also mixed quic-http3 vs HttpClient	Matched-client H3→H1 ≈ 0.87; do not prototype a second QUIC stack
H1→H2 / H3→H2 still ≪0.80 after pool	Cool A/B + c=1 + dumpasync/dotnet-trace @ c=32 (results/h2-origin-choke/)	Not dual-TLS polish: c=1 TWP faster (1.49×). Residual is outbound Http2OriginConnection.SendAsync queueing (TTFB≈SendAsync wait grows 624→2263 µs c=8→32; 13 parked SendAsync on H3→H2; 102 SemaphoreSlim TaskNodes on H1→H2; managed reverse peer only ~7 in-flight forwarders). Monitor slow-path ~2× managed reverse peer	Origin Http2FrameWriter exclusive drain: encode+enqueue under short writeLock, no WriteAsync under the lock (reference .NET server stack model). Cool H1→H2 0.87× @ c=32 (28,996 / 33,336, rps-ramp-20260818-170412/170452); H3→H2 0.64× @ c=32. TTFB p50 262→894 µs. See h2-origin-choke/POSTFIX.md
H1→H2 / H3→H2 still <0.80 after origin frame writer	Cool A/B + grow A/B + gcdump/trace (results/residual-sub08/)	Scaling wait on origin HEADERS (c=1 1.04×); grow 4→32 regresses; ForceRead/HPACK noise; Channel/Pipe not retained-heap	Ranked in residual-sub08/CONCLUSIONS.md
Monitor.Enter_Slowpath ~9.5% after frame writer	syncblk + speedscope + pool-pick diag; post-fix traces (POSTFIX.md)	~70% Monitor was TryPick + ConcurrentDictionary.Count under entry.Gate. c=32: 0% soft-miss. c=64: ~21% soft-miss + CreationGate at max	Shipped A+B+C: Interlocked ActiveStreamCount; skip CreationGate on Gate-held Count >= max; snapshot pick outside Gate. Monitor exclusive 9.5% → 3.1%. Phase C no further win. Long-window TWP @ c=32 unchanged (~28.7k). Residual still HEADERS fan-in
H1→H2 still ~0.71× after pool-pick; dumpasync showed ForceRead on origin ReadLoop	Cool remasure + code path (POSTFIX-INTAKE.md)	Origin ReadLoop still did ForceRead 9+payload and copied HEADERS to MemoryStream; DATA awaited BodyPipe on the loop	Shared Http2FrameIntake on origin + in-place END_HEADERS decode + sync BodyPipe write. ForceRead removed. Best long cool pair this session still 0.71× (thermally soft absolutes) — next dig is post-headers path, not another receive rewrite
Post-intake: is residual WriteResponse / SessionEventArgs / still HEADERS wait?	dumpasync + topN + gcdump + stage timing (POSTFIX-POST-HEADERS.md)	Soft box (IDE CPU); dumpasync: no ForceRead, no InterimChannel/SendAsync park — bridges on client ReadRequestLine, origin on FrameIntake.Fill. Stage: TTFB ~93% of total, delivery ~5%. Pooling gates not cleared	Wait shape fixed. Do not pool or rewrite H1 write yet. Need cool quiet remasure + high-RPS alloc/CPU sample before next code change
Quiet remasure after restart: does cool ratio move? Gate A/B at high RPS?	Cool pairs + dumpasync + AllocationTick (quiet-remeasure/QUIET-REMEASURE.md)	High perf: H1→H2 c=32 0.71× (31.9k/44.7k); c=64 0.87×. High-RPS dump: ForceRead/Interim park still 0. AllocTick: SessionEventArgs+HeaderCollection 4.5% (<5% Gate A). Interim channel arrays ~7%+ but gated behind A. Monitor exclusive ~2.5%	No library change. c=32 residual confirmed; pooling/write gates still not cleared. Optional: YARP twin AllocTick for asymmetry
YARP twin AllocTick + InterimChannel passthrough lite	Twin gc-verbose + remasure (INTERIM-LITE.md)	TWP ~3× AllocTicks/request vs YARP; Interim Channel/segment ~11% TWP-only. Lazy InterimChannel when on1xx null; H1→H2 passthrough skips relay when no interception	Landed. Soft post-lite pair 0.82× (26.8k/32.5k); cool High-perf confirm blocked by IDE CPU — remeasure on quiet box before publishing ≥0.80
Cool confirm after InterimChannel lite	Paired c=32 High perf (interim-lite-confirm/CONFIRM.md)	TWP 33.6k / YARP 44.0k = 0.76× (was 0.71× pre-lite). Phase-A-class absolutes	Lite helped (~+5–7% relative) but still ≪0.80. Next: TTFB residual dig on no-intercept path
Post-lite TTFB dig @ ~31k RPS	dumpasync --fields + topN (interim-lite-confirm/TTFB-DIG.md)	20 SendAsync on origin writeLock (Semaphore maxCount=1, streamOpened=false); 6 on HeadersReceived; InterimChannel still 0. Lite on1xx=null confirmed	Residual is writeLock stream-open convoy, not headers wait / WriteResponse. Next: shrink work under origin writeLock (HPACK encode+enqueue)
H3→H2 cool remasure + gap fix plan	Cool c=32 pair (h3h2-fresh/CONFIRM.md) + FIX-PLAN.md / canvas	H3→H2 0.70× (26.0k/36.9k) — wiki 0.33× stale. Same origin writeLock; H3 still always allocates InterimChannel	P0 H3 Interim lite → P1 shrink encode under writeLock → P2 H3 Via/prep trim → P3 remasure other H2 arms
P0+P1 bundle: H3 lite, Via skip, SoftStream=2, HPACK method cache	Cool High perf (post-p0p1/)	H1→H2 0.89× (37.3k/42.0k); soft confirm 0.82×. H3→H2 0.73× (24.9k/34.2k). Max-conn 16 aborted (soft regress)	H1→H2 c=32 bar closed. Continue H3→H2 (≥0.80) + remasure other H2 arms
Remeasure H2 TLS→h2c / h2c→h2c after intake+lite era	Cool High perf c=32 20s (passthrough-fresh/)	H2 TLS→h2c 0.78× (51.0k/65.8k); h2c→h2c 0.73× (49.7k/68.3k) — up from ~0.66/0.70 wiki	Still ≪0.80 on passthrough; next dig client FrameWriter/HPACK (not origin pool)
HPACK static GetIndex bug + encode under writeLock + scheme patch	Cool High perf (post-hpack-static/ + post-hpack-confirm/)	StaticTable.GetIndex(name,value) compared ByteString to string → never matched; EncodeHeaderBlock allocated new Uri under writeLock; mixed-transport scheme 0x86↔0x87 patch; SoftStream=1; skip Via on H2 response IsFastPath; skip NoOp HPACK decode on verbatim compressed relay	H2 TLS→h2c 0.81× (45.8k/56.2k) closed. H1→H2 0.85×. H3→H2 0.72×, h2c→h2c 0.74× still open
OriginRelayPool SoftCap 8→1/2 fan-out	Cool remasure (post-relay-soft1/2)	Soft=1/2 did not beat Soft≈8 on h2c→h2c (extra cleartext legs)	Reverted SoftCap formula; residual is not origin-leg count
H3→H2 dump @ 26k RPS + QPACK dict encode	dumpasync (h3-profile/) + QPACK O(1) static lookup	32/32 SendAsync on HeadersReceived (not writeLock); 8 origin ReadLoops. SoftStream fan-out already enough	Residual is H3 session/QPACK/bridge CPU, not origin write convoy. QPACK static dict + response header path trim shipped; cool ratio still ≪0.80 — next SessionEventArgs-lite / pool
H3 inbound ≪ H2 / ≪ YARP H3→H3	Cool YARP-first matrix + shape (h3-vs-h2/, h3-verbatim-fair/)	Full SessionEventArgs + response QPACK decode/re-encode on every H3→H3 GET; YARP cool H3→H3 ~26–28k while TWP sat ~20k (0.70×)	Session-lite for H3→H2/H3/H1 + verbatim origin→client H3 frame relay (H2 compressed-relay analogue). Cool H3→H3 1.14× YARP (29.6k / 26.0k); H3→H2 / H3→H1 ≥0.80; MITM÷cleartext 0.93
H3 bodiless fast path + PrepareH2 skip + EncodeResponse + compressed DATA→wire	Cool High perf (post-encode-response/)	Skip InterceptionContext; drain FIN without body-pump lambdas; skip PrepareH2 RemoveHeader scan on IsFastPath; QpackEncoder.EncodeResponse (no List); compressed-relay DATA ReadExact into rented wire buffer; ReturnPayload after QPACK decode	Absolutes up (H3→H2 31.7k/44.1k; h2c→h2c 65.7k/91.3k) but ratios still ~0.72×. Lazy BoundedBodyPipe aborted (empty-body race). Skip linked-CTS on fast path aborted (abort cancel lost → ~0.67×). Next: SessionEventArgs-lite / pool
H3→H2 SessionEventArgs-lite (H3H2FastForward)	Cool High perf (h3-lite-only/)	Skip entire session/HttpWebClient/Null stream/empty Response on interception-off bodiless H3→H2; keep Request for HPACK only	H3→H2 0.83× (33.2k/40.0k) closed. Lazy BodyPipe re-tried + aborted again (empty-body hang / ~856 RPS). h2c→h2c still 0.74× (56.7k/76.3k)
h2c ThreadPool IOCP floor + SoftCap32 / exclusive drain / sync cont	Cool High perf (h2c-iocp-min/, aborted siblings)	Profile: LowLevelLifoSemaphore wait ~46%. Mirror worker min onto IOCP; default worker floor ×8/64. SoftCap32 / exclusive FrameWriter / sync continuations / WINDOW_UPDATE enqueue / CTS TryReset all aborted (regress or hang)	h2c→h2c 0.76× (62.6k/82.5k). Still open vs ≥0.80
Compressed finalize sync + CTS TryReset (no SoftCap change)	Cool pairs (h2c-cts-reset2/, h2c-soft16/)	Skip PendingFinalizations Task on compressed; TryReset pooled CTS	TWP absolutes up to ~65.7k on soft YARP; cool YARP-first still ~0.75×. SoftCap floor 16 neutral; SoftCap 32 convoy
Compressed END_STREAM skip force-flush WINDOW_UPDATE	Cool sandwich (h2c-wu-batch-confirm/)	Tiny-GET forced connection WINDOW_UPDATE pair per ~56 B response (~6% GrantReceiveCredit); batch connection credit to 384 KiB, drop stream credit on close	h2c→h2c 1.20× (102.8k/85.9k) closed — TWP leads. TWP A/B both ~102k
Full reverse+MITM cool audit + scheme decode-free + H2→H1 bridge trim	Cool High perf (gap-audit/, gap-audit-yarp/, h2h1-post-cts/, h2h1-vs-h2h2/)	Wiki h2c→H2 TLS 0.57× / MITM H3→H2 3.5k stale. Mixed-transport :scheme skips Decoder when 0x86↔0x87 patch works. H2→H1: buffer tiny bodies, skip Linked CTS / framing validator / empty Before* on IsFastPath; struct enumerator for LowercaseHeaderNames. New reverse-http1-mitm fair twin	h2c→H2 TLS 1.29× (89k/69k). H1 plain 1.19×. H1 TLS 0.91×. Transparent H1 MITM÷terminate 0.96×. H2→H2 MITM 104k. H2→H1 MITM÷cleartext 0.82–0.86× residual (per-stream H1 fan-out × dual TLS; H2→H2 dual-TLS proves TLS not the choke)
Origin HTTPS: unwrap nested HttpServerStream + skip SslStream header flush	Cool (ssl-unwrap2/) + dump diagnosis	Direct HTTPS was NetworkStream→HttpServerStream→SslStream→HttpServerStream; header Write flushed every SslStream. Finish() no-op when Content-Length exact	Correctness/cleanup landed; cool MITM÷cleartext still ~0.84× (47.6k/56.8k). Residual not nesting/flush
H2→H1 MITM÷cleartext ≪0.90 after ssl-unwrap	Cool 3/5-rep High-perf (h2h1-hostcache/, h2h1-gate12/, h2h1-final/, h2h1-harden/) + thread-time sample	Whole-RT SoftCap serialized warm SslStream reuse; DataAvailable closed keep-alives after exact-CL reads; per-request pool-key StringBuilder; cert callback .Wait on CompletedTask	Create-only SoftCap (MaxConcurrentHttp11HttpsOriginCreates, default Clamp(ProcessorCount,4,32)); always close on residual DataAvailable (no IsFastPath skip); cache CachedHttp11PoolKey; sync cert validation. Cool 5-rep median 0.90 / mean 0.92 (h2h1-final/); harden 3-rep median 1.03 / mean 0.91 (h2h1-harden/). Lock-free pool rent aborted (noisy dips)
H1→H2 still ≪0.80 after pool + named session micro-opts	Cool matched A/B (matched-post-fix)	Dual client+origin TLS + per-request session on an 8-thread box; dumpasync already showed multiple origin ReadLoops (not a single-conn convoy)	Superseded by h2-origin-choke/ (2026-08-18): see row above
H3→H2 c=8/16 lost ~30–40% vs exclusive-bag after pooling	Cool A/B (profile-baseline) + dumpasync/dotnet-trace @ c=16 (h3h2-c16.dmp / .nettrace)	Grow threshold 16 pinned all streams on one origin ReadLoopAsync; 716 SemaphoreSlim waiters; H3 GET also did HEADERS + empty DATA	Grow at 4 active streams + drain FIN then HEADERS+END_STREAM for bodiless H3. Recovered 8,418 @ c=16 (profile-post-fix, vs phase-0 8,539)
H2 TLS→h2c / h2c→h2c ~0.63–0.66× cool	dumpasync + sampled trace @ c=32	Http2FrameWriter already on DATA path; ForceRead per frame header; HEADERS still two WriteAsync under the lock	Large-read Http2FrameIntake (64 KiB) + enqueue stream-scoped HEADERS on Http2FrameWriter. Matched cool h2c→h2c ≈ 0.70, H2 TLS→h2c ≈ 0.66 (matched-post-headers-writer)
Cool H3→H1 ~0.36× peer (12.1k / 33.4k)	Cool pair + trace @ c=32	Invalid ratio: TWP quic-http3 vs peer HttpClient. Trace was session/HandleAsync, not MsQuic-native	Match clients; later dual-listen reverse H3 enables HttpClient both sides (matched-httpclient-h3/, H3→H1 ≈ 0.87)
H3→H1 integ empty body; Windows ~0.79× YARP	DualListen / ForcedHttp11Origin + cool pair	Fast path buffered only known Content-Length; Kestrel WriteAsync often chunked → body never drained before pool Release; H1 Title-Case names paid QPACK ToLower every response	Drain chunked/connection-close via LimitedStream before Release; NormalizeNamesToLowerAscii + HeaderNamesAreHttp2Normalized; decode H2 HEADERS into Response headers (no second collection). Cool Windows H3→H1 ≈ 0.96×, H3→H2 ≈ 1.06×; Linux H3→H1/H2/H3 ≈ 1.04× / 1.15× / 1.20× (32552296839, 32552295495)
H1→H3 “name normalize + tiny-body coalesce” looked like H1→H2 gap	Cool A/B (win-parity-audit-20260822-*)	Hypothesis: Title-Case QPACK tax + missing H1→H2-style fast commit / TLS coalesce	Name-normalize path fully reverted (~1.13× → 0.65×). Eager-buffer alone also poisoned the pool — see next row.
H1→H3 Win CI ~0.94×; H1 client + StreamBodyWriter = header-only TLS record	Cool A/B + dispose/RST dig (cool-h3-origin-eager64-drain-20260823/)	Known-CL ≤64 KiB H3 origin bodies streamed via StreamBodyWriter → H1 WriteResponseAsync then body (same class as lossy H1). First eager-buffer attempt disposed the Quic stream before FIN → RST / pool poison (1.16×→0.7×)	Eager-buffer ≤64 KiB and drain frames to FIN before DisposeAsync in ForwardOverQuicAsync. Cool TY/YT ≈ 1.23× / 1.03×. CI remasure bridges next.
H2→H1 64 KiB ~0.87× YARP (tiny-GET already parity)	Cool pair + code compare vs Kestrel/YARP	Streamed path stripped Content-Length then empty END_STREAM DATA; pump wrote 8 KiB fills → 8 DATA frames + trailer; HttpStream double-buffered socket→8 KiB→dest; QueueDataFrame + 32 KiB flatten	Keep CL + END_STREAM on last DATA; HttpStream large-read bypass; in-place DATA framing (flatten kept); skip LimitedStream/Via on known-CL fast path; raise flatten budget to 288 KiB. Cool 64 KiB ≈ 1.13×; 256 KiB ≈ 0.89×. Dropping flatten alone still ~0.65×
H2 POST cool ~0.88× / 256 KiB H2→H1 ~0.90×	Shape c=1 vs c=32 + YARP StreamCopier (64 KiB) compare	c=1 TWP leads POST (~1.2×); c=32 loses when YARP healthy — multiplex tax (frame-loop copy + shared client writer). Extra body memcpy / coalesce experiments	Kept: ArrayPool request-body channel + TryReserve on CopyFromAsync. Do not: reserve >1 frame before enqueue; slice control frames into coalesced DATA; drop flatten
H3 early-response Win CI ~0.76× (Linux already ~1.02×)	Cool A/B (fix-early-tls/) + origin/YARP duplex compare	ForwardOverTcpAsync wrote the full request body before ReceiveResponse while the probe origin overlaps after 8 KiB (YARP StreamCopier same). H3+MsQuic amplifies the serialization on Windows	Overlap streamed upload with ReceiveResponse; fold remaining upload into StreamBodyWriter via Task.WhenAll. Cool mean ≈ 1.21× YARP. Do not re-land Http3Frame coalesce 256→16 KiB (hurt POST)
H3→H1 64 KiB GET Win CI ~0.56× / Linux ~0.82×	Cool A/B (h3-64k-rebaseline/) + CI remasure	Cool mean ≈ 1.13× (3118/2688 & 3488/3181); stale CI was pre-StreamBodyWriter	No library change. Publishable 32611185635 @ cd276c83: Win ≈ 1.15× (3,752 / 3,269), Linux ≈ 1.25× (5,295 / 4,247). Next body gap: Win H1 TLS 256 KiB ≈ 0.85×
H1 TLS→H1 256 KiB Win CI ~0.85×	Cool A/B + shape (h1-256k-cool/) + YARP StreamCopier compare	Cool c=1 ≈ 0.83× (per-request); CopyBytesToStream FillBuffer’d 8 KiB forever — H2 large-read bypass never ran on H1 known-CL copy	Rent 64 KiB + ReadAsync when parser window empty (HttpStream.CopyBytesToStream, 106e73b9). Cool c=1 ≈ 1.16×, c=32 ≈ 1.09×. Publishable 32614286032: Win ≈ 1.12× (2,617 / 2,347).
H1 TLS new-conn Win CI ~0.84x (Linux TWP leads)	Cool A/B + Kestrel SocketConnectionListener / ConnectionDispatcher + bare ceiling	Nested SslStream + ClientHello peek + ECDSA + Task.Run + BeginAccept APM + per-accept linger/timeouts + RetryPolicy closures; lite path forwarded Connection: close to origin → no origin pool under NC	Peek/unwrap/RSA/no-keepalive; abortive SO_LINGER(0) on close; AcceptAsync; CTS pool; session-lite; WaitForData-before-SslStream; 8 KiB rent. AcceptIOQueue no win. Bare NC Connection: close response-skip fixed. Strip hop-by-hop Connection before origin write on H1 terminate lite. Publishable 32625349927 @ 13059143: Win NC ≈ 1.01×, Linux NC ≈ 1.01× YARP (nginx 1st on Linux NC — TWP 2nd).
H1→H3 100% err after session-lite (03159694)	Bisect 11e32f1c→03159694 + curl serve	H1 terminate lite gated only on ForwardHost + bodiless GET — H1→H3/H2 with forced upstream H3/H2 took TCP H1 lite against QUIC/h2-only origins	Skip session-lite when connection-level UpstreamHttpProtocol is Http2/Http3 (62e5efcd). Soft coolish H1→H3 ≈ 1.25×, h2c→H1 ≈ 1.05×, H3→H1 ≈ 1.09×.
WarmTls H1→H3 + CachedServerAuthOptions gate broke H2 reverse	Local lossy H2 + ALPN fail (No common application protocol)	Expanding fixed-cert to any warmed CachedServerAuthOptions pinned http/1.1-only ALPN while H2 clients offer h2	Gate fixed-cert on !EnableHttp2 only (H3 clients use QuicListener). H1→H3 host: EnableHttp2=false + WarmTls (8ac422ee). Cool H1→H3 ≈ 1.16×; remasure bridges/bodies/lossy @ tip.
H2/H3→H1 64 KiB / lossy H2 still tax many DATA fills	Code compare vs H1 ≤64 KiB coalesce	H2→H1 / H3→H1 eager-buffer capped at 16 KiB while lossy/bodies GET is 64 KiB → stream via ~4× 16 KiB fills (shim delayMs per read)	Raise eager known-CL threshold to 64 KiB (Http2ToHttp11BridgeHandler + Http3OriginBridge, 8ac422ee). CI bodies Win H3→H1 64 KiB ≈ 1.09× (32631121563).
H3→H1 tiny Win CI ~0.90×; cool order-noisy ~0.94–1.02×	Cool both-order dig + HEADERS+DATA coalesce experiment	SendResponseAsync does separate QuicStream writes for HEADERS then DATA + Flush	Reverted single-write HEADERS+DATA coalesce (cool-h3-headers-data-coalesce-20260823/): cool mean ≈ 0.96× (no win).
H3→H1 c=1 leads ~1.25×; c=32 loses (~0.96×) — multiplex shape	Cool c=1 both orders + dumpasync @ c=32 (dig-h3h1-dump2-20260823/) + SampleProfiler	32/32 parked on origin ReceiveResponse→FillBuffer (not writeLock/SoftCap). CPU: both TWP and YARP ~60% LowLevelLifoSemaphore wait — not a unique ThreadPool starve. QPACK/Normalize ≪1% exclusive	Next: cycle-time after status line (body buffer → QPACK → Quic write → pool Release) vs YARP HttpForwarder; optional AllocTick asymmetry. Soft remasure @ 8e5c181b (cool-h3h1-shape-20260823/): c=1 ≈ 1.60×, c=32 ≈ 1.07× — CI Win still ~0.89× @ prior tip; remasure bridges in flight.
H3→H1 early origin Release / skip client Flush before CompleteWrites	Cool A/B (cool-h3h1-early-release-20260823/, cool-h3h1-skip-flush-20260823/)	Hypothesis: free H1 socket sooner / avoid MsQuic Flush tax on fast path	No cool win (~0.95–0.98×). Reverted both.
H1→H3 / h2c→H3 Win CI closed @ 8789d6de	CI bridges 32636039240	Eager ≤64 KiB H3 origin body + FIN drain before Dispose	Win H1→H3 ≈ 1.04×, h2c→H3 ≈ 1.05×. Still open: Win h2c→H1 ≈ 0.96×, H3→H1 ≈ 0.89×; lossy H2 16 vs 17.
H3→H1 ForwardHost Host rewrite (match YARP HttpForwarder)	Cool A/B (cool-h3h1-host-rewrite-20260823/)	Hypothesis: :authority localhost:<listen> on H1 Host wire vs origin 127.0.0.1:<port>	No cool win (~0.94–0.99×). Reverted.
h2c→H1 early origin Release before EmitSynthetic	Cool lead ~1.03×; CI remasure 32638840153 @ 253e8716	Hypothesis: free H1 socket before H2 frame emit	CI miss: Win h2c→H1 still ~0.95×; Lin h2c→H1 regressed ~1.03×→0.96×. Reverted.
h2c→H1 ForwardHost Host rewrite	Cool A/B (cool-h2ch1-fwdhost2-20260823/)	Same Host rewrite on H2→H1 bridge	No clear cool win (YT ~0.97×). Reverted.
Lossy H1 Win remasure @ 253e8716 ≈ 1.00× (662/662)	CI lossy 32638842839	Prior ~0.99× noise	Closed ≥1.00× (nginx 1st 634 — TWP 2nd). Win lossy H2 still 16 vs 17.
Lossy H2 HOL: NullOriginStream lacked SETTINGS_MAX_CONCURRENT_STREAMS	Cool + CI lossy 32643126466; bridges tax @ same tip	Hardcoding MaxStreams=8 on NullOriginStream closed lossy (Win 3.47×) but tax'd Win tiny-GET h2c→H1 / H2→H1 (~0.90×) via extra TCP handshakes at c=64.	Landed lossy-only: probe sets ResourceLimits.MaxConcurrentStreamsPerConnection=8 when IsLossy (WithMaxConcurrentStreams…); Http2Helper appends SETTINGS. Tiny-GET keeps default 256. Cool lossy 29/15; cool h2c ≈ 1.04×. Remasure lossy+bridges. Dispose harden on TcpClientConnection kept.
Lossy H1 Win cool ~0.86× (p50 +16 ms vs YARP)	Cool A/B + userspace delay shim analysis	Fast-path WriteResponse then CopyBody emitted a header-only TLS record before body; shim pays delayMs per read → ~3 extra 5 ms trips	Materialize known-CL ≤64 KiB on fast path + coalesce headers+body (bc768069). Cool ≈ 1.00×; CI 32620889168: Win 663/664, Linux 1199/1196.
GHA compare-post/compare-arch failed; laptop H3 POST/slow passed	Failed run logs (32602145518, 32602146550)	(1) Dual-listen: TCP ephemeral then QUIC UDP same port → Windows WSAEADDRINUSE when UDP busy/excluded. (2) Incomplete StreamBodyWriter + DataAvailable==0 pooled origin sockets with unread CL → next request H3_INTERNAL_ERROR (HeadersRead slow-consumer + warmup cancel amplifies on 4 vCPU)	Retry ephemeral TCP+QUIC bind in ProxyServer.Start/AddEndPoint; always close origin on incomplete StreamBodyWriter; YARP/nginx dual-stack free-port pick
H3→H1 latency bundle (skip drain / skip Flush / HEADERS+DATA coalesce)	Cool absolute win (cool-h3h1-latency-bundle-20260823/) + CI bridges 32652931261 @ 3f948409	Cool c=64 TWP ~25k (tip ~20–23k); laptop YARP ~30k → cool TY ~0.85×. Trace @ c=64: p50 gap not exclusive CPU.	CI miss: Win H3→H1 0.92× → ~0.87× (13,391 / 15,444). Lin H3→H1 still leads (~1.11×). Lossy Lin H3 improved 0.76×→0.89× (278/314) but still <1.00×. Reverted (2bf18d75).
H2→H1 Memory ~5–9× YARP at RPS parity	Saturation RSS sampler + bag lifetime analysis	ConcurrentBag<Task> PendingSynthetics/Finalizations retained completed Tasks (session closures) for client H2 conn life; full SessionEventArgs per stream on IsFastPath	Http2PendingWork remove-on-complete; H2→H1 warm TryRentPooled + HeaderBuilder wire (H3→H1 analogue). Remeasure Block B Memory. Do not shrink windows / single-conn.
H3→H1 Win ~0.993× residual after one-pass QPACK	Cool A/B + gen0 on post-status path	Per-response MemoryStream QPACK builder + Latin-1 string round-trip + new byte[] tiny body	ThreadStatic ResponseBlockBuilder rent; span AddHeader; BufferPool body when Available covers CL. Gate ≥1.00× on bridges CI.
H3→H1 skip Normalize on tiny (b54f5708)	CI bridges remasure	Hypothesis: avoid Clear+new HttpHeader per field; EncodeResponse ToLowerInvariant instead	CI miss: Win H3→H1 0.95×→0.92×. Reverted on develop (always Normalize again).
H3→H1 sticky ConcurrentBag TCP pool (bypass factory poolLock)	Cool tip-vs-sticky both orders @ c=32 (cool-sticky-vs-tip-20260823/)	Hypothesis: CI 4 vCPU multiplex tax on poolLock/queue	Cool flat (~1.07× both). Reverted.
H3→H1 Request/Response ConcurrentBag + QPACK ArrayPool EncodeResponse	Cool same-thermal tip-vs-pool @ c=32 (cool-pool-qpack-20260823/)	Hypothesis: HeaderCollection graph + QPACK byte[] gen0 under 32 wakeups	Cool flat (tip ~1.08×, pool ~1.06×). TWP absolute sometimes +high single digits; ratio not ≥+2%. Reverted.
H3→H1 FIN-drain overlap with origin TCP + message pool	Cool both orders (cool-pool-overlap-20260823/)	Hypothesis: hide Quic FIN read behind origin FillBuffer (do not skip drain)	Cool miss (order-noisy mean ≪ tip). Reverted.
H3→H1 one-pass H1 headers → QPACK (no Response/HeaderCollection)	Cool tip-vs-onepass YT @ c=32 (cool-onepass-v2-20260823/) + H3 integ	Post-status HeaderCollection graph + EncodeResponse under 32 wakeups (CI 4 vCPU gen0)	Cool YT win: one-pass TWP÷YARP ≈ 1.24× (20,893/16,904) vs tip ≈ 1.18× (19,807/16,843); TWP absolute +~5%. TY polluted by soft YARP. Unit+H3 integ green.

Guardrails while optimizing

• Real proxy improvements only — do not game the RPS harness. YARP (and nginx) are the yardstick, not the product. Land changes that reduce real work on the hot path for general reverse-proxy / keep-alive traffic: fewer allocations, fewer syscalls, less protocol waste, less interception tax when unused. Do not land knobs, special cases, or architecture copies whose only purpose is to inflate TWP÷YARP on the tiny-GET probe. Use the probe to find where time burns; revert experiments that do not help the proxy itself even if a noisy pair looks green.
• Full unit + integration suites after every change. Several perf changes introduced real regressions (the scheme mismatch, the DATA race, HTTP-version and Content-Length bugs on the bridges); the suites and the benchmark's own error SLO caught all of them the same day.
• A standalone external repro (tools/H2ExternalRepro) validates against real internet sites, which surface flow-control and settings behavior loopback benchmarks never exercise.
• Wiki numbers carry their run IDs and an explanation of why each number moved, so a future regression has a baseline with provenance.

Checklist

1. Re-baseline with paired same-thermal A/B before believing any gap.
2. Sweep concurrency — let the curve's shape choose the tool (serialization → dumps; per-request cost → CPU sampling).
3. dumpasync for where requests wait; dotnet-trace for where cycles burn.
4. Decompose internal vs client-observed latency (TWP_RPS_STAGE_TIMING); a large gap means queueing upstream of the pipeline.
5. Read the faster system's source to answer named hypotheses; keep TWP's architecture.
6. Before keeping a change: confirm it is a real proxy improvement (less work / alloc / I/O on a general hot path), not a probe-only tweak to beat YARP. For Memory lites: keep only if RSS improves (or not worse) and RPS ÷YARP does not regress.
7. For Memory: use harness proxy_rss_peak_bytes / descendant-tree sampler; confirm retention with gcdump when bags/sessions look sticky; do not shrink H2 windows or force single-connection to game RSS.
8. Run the full test suites and the external repro before publishing; record run IDs in the wiki.