feat(PR-A3c): batched multi-tenant parallel decode bench (per-session binding, recall-preserving S5) — awaiting GPU run

[FluffyAIcode]

What

PR-A3c direction realized: per-session binding + true parallel multi-tenant decode, recall-preserving (S5 only — recall is the bottom line, non-recall configs out of scope). On a single accelerator, "parallel" = a batched forward where each batch row is a session with its own KV-cache row. scripts/research/k3_cuda_multitenant_parallel_bench.py runs batched AR vs batched restored-S5 at N=1..8 on the same prompts.

Result (H200 NVL, gemma-4-26B-A4B 4-bit, NIAH ctx≈1238) ✅ RAN

sessions N	restored-S5 agg tok/s	parallel speedup vs N=1	per-session recall	peak
1	27.4	1.00×	1.0	57.6 GB
2	54.6	1.99×	1.0	60.4 GB
4	111.6	4.07×	1.0	66.0 GB
8	220.4	8.04×	1.0	77.3 GB

• Near-linear parallel scaling (8.04× at N=8) — the engine genuinely decodes N sessions in parallel, the opposite of v0.3's single-tenant serialized path (where concurrent sessions serialize, latency linear in N).
• Per-session recall 1.0 at every batch size — recall preserved under batched multi-tenant decode (bottom line met).
• Restored S5 ≈ native AR (220.4 vs 216.4 tok/s @ N=8) — restoration is free, and bounded memory (ADR 0014 §3.4: ~4× more agents fit). PR-A3c delivers parallel throughput + bounded memory + recall together.

Changes

• scripts/research/k3_cuda_multitenant_parallel_bench.py (new bench)
• inference_engine/v04/restored_attention.py — allow batch-1 RoPE cos/sin to broadcast over B>1 (the one batch-1 fix needed; 32 existing RoPE tests still pass)
• docs/adr/0014-…md §3.5 + Appendix A; results/research/k3_cuda_multitenant_parallel_gpu.json

Scope / follow-ups

• This validates the engine/batched-decode per-session-binding capability. Wiring it into the gRPC served path (SessionStore → per-session adapter + batched scheduler) is the remaining productization step. Batched fused spec-decode (DFlash is batch-1 today) is a separate follow-up.

Testing

• ✅ pytest tests/inference_engine/v04/test_restored_attention.py (32 passed — RoPE broadcast fix safe)
• ✅ Ran on H200 NVL: 8.04× scaling at N=8, per-session recall 1.0, restored ≈ AR.