PR-E1b (ADR 0008 §6.5): gRPC long-session bench + server CLI

[FluffyAIcode]

Why

PR-E1's deferred sibling. Ships everything needed to validate the two ADR 0008 §7 GA gates the deprecated HTTP shim's bench cannot answer:

Gate	What it checks
memory bounded	agg.kv_bounded is True (KV stays inside a tight band)
prefill bounded	agg.prefill_bounded is True (per-turn latency is flat across the run)

The HTTP shim's bench_long_session.py fails on prefill-bounded by architecture — every /v1/chat/completions request re-prefills the full conversation history. The session-bound gRPC contract makes prefill cost depend only on the size of each new user message, regardless of conversation length. PR-E1b measures this empirically.

Mac M4 4-hour evidence — GA gate G2 PASSED ✅

Committed to main as bec3d7b: results/platform-tests/bench_session_4h_1780332893.json

=== Run config ===
  duration_s         = 14400.01
  turns              = 480           (24 × 10-min buckets, 20 turns/bucket — perfect cadence)
  errors             = 0
  abort_reason       = None
  partial            = false

=== Headline KPIs ===
  p50 latency        = 1.8286 s
  p95 latency        = 1.8529 s
  p95 - p50          = 0.024 s        (latency distribution is essentially a delta function)
  latency_drift_p50_s= 0.0093 s       (9 ms drift over 14400 s)
  bucket-p50 spread  = 0.014 s        (1.824s head → 1.838s tail)
  prefill_bounded    = True
  kv_bounded         = True           (see caveat below)

Comparison to v0.3.0-rc1 HTTP-shim 4h run

Metric	v0.3.0-rc1 (HTTP shim)	v0.3 PR-E1b (gRPC session)	Δ
n_turns	58	480	8.3×
n_errors	many (timeout loop)	0	—
latency_drift_p50_s	+39.74 s	+0.0093 s	4400× improvement

This is the empirical proof that the session-bound gRPC contract delivers the prefill-bounded property the deprecated HTTP shim could not, and is the binding evidence for ADR 0008 §7 GA gate G2.

Caveat: kv_live_bytes is always 0 in this report

The bench reports min/mean/max kv_live_bytes = 0 across all 480 turns. The verifier's KV cache is not actually empty — this is a reporting-path issue:

• GetSessionInfo.kv_live_bytes reads Session.slab.live_kv_bytes (PR-A3b's wiring).
• KVSlab.live_kv_bytes reflects whether anyone has called slab.append() to write K/V tensors into the slab.
• The verifier maintains its own SinkWindowKVCache (an mlx_lm._BaseCache subclass) and never writes through to the slab.
• The slab is therefore an "this session holds one capacity unit" placeholder, not a true byte gauge.

kv_bounded = True therefore reduces to 0 - 0 < 10% × max(0,1) = 0.10 — trivially True from a constant-zero series. Architecturally kv_bounded is a mathematical guarantee of the SinkWindowKVCache construction (capacity = (sink + window) tokens fixed at build time, here 68 tokens). 4h × 480 turns × no OOM / no crash is the indirect empirical confirmation. But the bench's headline number is degenerate.

Fix queued as PR-E1c: wire GetSessionInfo.kv_live_bytes to the verifier's cache.k_seq_length × num_layers × num_kv_heads × head_dim × bytes_per_dtype × 2 (K + V). ~50-100 lines + unit tests. Independent of merging this PR.

What ships

inference_engine/bench/ (new package, 100% covered)

File	Stmts	Purpose
session_long_run.py	56	Pure aggregator: _percentile, _kv_bounded, _prefill_bounded, _latency_drift_p50_s, _bucketize_10min, aggregate_run. No numpy dep.

scripts/start_grpc_runtime_server.py (new, CLI plumbing)

Boots a real Qwen3 verifier (CPU or MLX), wires SessionStore + AppendTokensCoordinator + GenerationCoordinator, serves the gRPC RuntimeService. Pulls slab dims (num_layers, num_kv_heads, head_dim) from the verifier's HF config.

scripts/bench_agentic/bench_session_long_run.py (new, CLI plumbing)

Walks ONE gRPC session through many turns. Tokenizes only the new user message per turn. Records latency + session.info().kv_live_bytes + history length. Atomic JSON via os.replace + 10-min partial checkpoints.

tests/inference_engine/bench/test_session_long_run.py (new)

35 tests covering the aggregator to 100%.

scripts/review_pr_e1b_on_mac.sh (new, executable)

Default = 30-min smoke. --print-server-cmd / --print-4h-cmd subcommands print bare commands for separate-terminal long runs.

.github/workflows/ci.yaml

• tests/inference_engine/bench/ added to the Linux pytest gate
• --cov=inference_engine.bench added to the coverage gate

Linux verification

Linux CI gate (extended path set + new bench tests):
  730 passed, 100% coverage on 1750 stmts.

CLI scripts:
  py_compile passes on both. End-to-end runs validated on Mac M4 (see above).

Stack

PR-E1b is independent of PR-D1 (#49) and PR-E1 (#50) at the file level. Branched off main directly. Can merge in any order with the others.

Reviewer checklist

• 4-hour Mac M4 evidence committed to main — results/platform-tests/bench_session_4h_1780332893.json (commit bec3d7b). prefill_bounded=True with 9 ms drift over 14400 s. ✅
• 0 errors across 480 turns. ✅
• Linux CI 100% coverage on the aggregator. ✅
• kv_live_bytes=0 caveat acknowledged; PR-E1c queued to fix the reporting wiring.

Next PR

• PR-E1c (queued, ~50-100 lines): wire GetSessionInfo.kv_live_bytes to the verifier's true cache byte count.
• PR-D2: HTTP shim refactor onto SessionStore. Independent of this PR.
• PR-E2: Self-hosted Mac M4 GitHub Actions workflow.