Server: fused single-GPU decode for --batched-session (co-decode 2-4 sessions in one graph pass)

[iCreil]

--batched-session N works beautifully on multi-GPU placements, where the
session pipeline fills different pipeline stages across devices. On a single
GPU I ran into two things while testing it on an RTX PRO 6000 (96 GB) with
the Flash q2 GGUF, and this PR is an attempt at addressing both. Happy to
rework any part of it — the branch is organized so the pieces can be taken
(or dropped) independently.

1. Resident session creation fails at normal context sizes

On 96 GB, --batched-session 4 --ctx 32768 (and even --batched-session 1 --ctx 32768) fails with failed to create cuda session: the lazy Q8→F16
weight caches warm up during engine open and grow until only the default
reserve (~1% of VRAM) stays free, and the resident sessions are created
after that point. Measured on q2: engine base lands at ~95.9 GiB while each
32k session only needs ~0.6 GiB of context buffers.

Fix (2 commits): the server passes the resident session count to the engine
(resident_sessions config field), and engine open computes a cache reserve
from the per-session ds4_context_memory estimate right before
accelerator_cache_model_tensors() — the estimate needs the model shape, so
it has to run inside open, not before. An explicit
DS4_CUDA_Q8_F16_CACHE_RESERVE_MB still wins. With this, --batched-session 4 --ctx 32768 --prefill-chunk 1024 boots at 97.2/97.9 GiB and single-stream
speed is unchanged (the hot part of the cache still fits).

2. Single-GPU decode batches don't scale

Without a placement, ds4_sessions_eval_batch_cuda encodes the one-token
graphs back to back. That preserves exactness, but each session re-reads the
full weight stream, so decode is memory-bound at 1x: measured 62 tok/s
aggregate at 2 and 4 concurrent clients vs 61 tok/s for one client.

This PR adds a fused single-tier path to the same function (the single-GPU
counterpart of metal_graph_encode_session_pipeline_batch): the dense
projections run once on the shared batch workspace with n_tokens = count,
so the weights cross memory once per step; KV append and compressor/indexer
bookkeeping run per sequence with the same single-token helpers the decode
layer uses; attention runs as one multi-sequence launch with per-sequence
fallbacks (indexed path, score-buffer overflow), so there is no position
limit.

Numbers (RTX PRO 6000, Flash q2, --batched-session 4 --ctx 32768 --prefill-chunk 1024, 200 generated tokens per request, aggregate =
total tokens / wall):

concurrent clients	main (one-token pipelining)	this PR (fused)
1	61	61 (same path)
2	62	68
4	could not boot / 62 at smaller ctx	85.6

Commit walkthrough

CUDA kernels (additive):

• CUDA: add multi-sequence decode attention kernel — one query token per
  sequence, each attending to its own KV ring/compressed cache
  (ds4_gpu_attn_seqview, by-value in kernel params, max 4).
• CUDA: add pos-vector RoPE tail variants — decode tokens of different
  sequences sit at arbitrary positions; per-element math identical to the
  single-token kernels (token t matches a pos0=pos[t] launch bitwise).
• CUDA: narrow-batch q8_0 matmul for 2-4 token decode — same idea as your
  tok2 kernel, generalized to n_tok 2-4 with n_tok as a parameter: one
  warp per output row, each 32-weight block read once and dotted against
  all tokens. Dispatched ahead of cuBLAS/MMA for n_tok 2-4
  (DS4_CUDA_NO_Q8_NARROW=1 restores the previous dispatch; the GLM verify
  force_decode_warp path is untouched).

Graph driver:

• metal_graph_decode_compressor_indexer — the per-token compressor/indexer
  segment of the decode layer phase, taken as a helper so the batched driver
  can run it per sequence on batch-scratch row views. It is a copy (the
  single-token path is untouched); if you'd rather have the phase function
  call the helper too to avoid the duplication, glad to do that instead.
• metal_graph_encode_layer_attention_decode_multi +
  metal_graph_encode_decode_multi — the fused pass. FFN and output head
  reuse the existing batch functions; logits land one row per sequence in
  spec_logits.
• Graph: run fused-decode routed experts per sequence — at batch 2-4 the
  selected experts of different sequences are almost always disjoint, so the
  batch expert path costs more than n single-token passes. This turned out
  to matter twice: without it the fused path was slower than pipelining (54
  vs 62) and one of two concurrent greedy streams diverged from the solo
  run; with it, both concurrent greedy streams match the solo stream
  token-for-token on the prompts I tested. DS4_FUSED_BATCH_MOE=1 keeps the
  batch expert path for comparison.

Server/glue:

• The fused path plugs into ds4_sessions_eval_batch_cuda for single-tier,
  non-streaming, non-distributed decode batches of 2-4;
  DS4_NO_FUSED_SESSION_BATCH=1 restores the one-token pipelining (and was
  used for the A/B numbers above).

Correctness notes and limitations

• The fused pass is not bitwise-identical to serial decode (batched matmul
  reduction order); greedy token streams matched the solo run in every check
  I ran (q2, 150-200 tokens, 2 and 4 concurrent). The kill switch makes the
  comparison easy to reproduce, and I kept it default-on only because the
  numbers looked convincing — happy to flip the default if you prefer
  opt-in.
• CUDA only: the multi-sequence kernels have no Metal twins, so Metal keeps
  its current paths untouched.
• SSD streaming sessions are excluded from the fused path (they fall back to
  the existing behavior).
• spec_logits is now allocated unless the fused path is disabled (~8 MiB
  per graph), since the fused output head lands there.

Test setup details, exact configs and raw runs are in the commit messages;
I can share the bench harness if useful. And if you had a different
direction in mind for single-GPU batching, happy to adapt this or just take
whatever pieces are useful.