CUDA: resident expert cache and faster selected-expert uploads for --ssd-streaming decode

[iCreil]

While testing --ssd-streaming decode on CUDA (RTX PRO 6000, 96 GB, the
q4 Flash GGUF served from tmpfs) I measured a flat ~2.3 tok/s no matter
how warm the page cache is: every decode step re-uploads all selected
experts of every layer from the host map. The backend rewrite left the
CUDA resident expert cache API as stubs (configured_count() returns 0,
the seed hooks are no-ops), so only the per-layer staging buffers exist.
This PR restores the resident cache and then chips away at what the
timings said was left. Warm single-stream decode goes from ~2.3 to
29-32 tok/s; the four commits are independent steps and each one is
A/B-able on the same binary via env switches. Happy to rework any part.

1. Restore the resident expert cache (~2.3 -> 16.7 tok/s)

Brings back a VRAM LRU cache for streamed experts in front of the
staging buffers: hits become device-to-device copies, misses load the
expert into the cache once. Compared to the pre-rewrite cache this
version also:

• keeps one cache per (gate,down) byte-size class (4 classes):
  mixed-quant GGUFs interleave layers with different expert byte sizes,
  and a single cache keyed on the last-seen size is released and
  rebuilt on every size transition, never accumulating hits;
• lets prefill batch loads read the cache (hits become d2d copies) and
  append into free capacity, but never evict: a long prompt cannot
  cycle out the warm decode working set. Decode loads evict normally;
• allocates the staging buffers with the cache still in place and only
  sacrifices it (once, with a retry) when VRAM is genuinely short;
• implements the existing prefill-seed/hotlist hooks on top of the same
  cache, and only reports a budget to the shared working-set estimates
  when the operator opted in (budget flag or env), so default runs keep
  the current planner behavior.

Budget comes from the existing --ssd-streaming-cache-experts /
DS4_CUDA_STREAMING_EXPERT_CACHE_N, live-capped against free VRAM with
a reserve (DS4_CUDA_STREAMING_EXPERT_CACHE_RESERVE_GB, default 16).
DS4_CUDA_STREAMING_EXPERT_CACHE_N=0 disables the cache (used for the
A/B numbers).

2. Skip the per-load VRAM query once the cache has settled (16.7 -> 23.5)

Host-side phase timing showed ~90% of the steady-state selected load
(~920us of ~1010us) inside cache prepare: when the VRAM-capped capacity
lands below the requested budget (e.g. 7000 requested, 5892 fitting),
the capacity >= target fast path can never hit and every load of
every layer falls through to cudaMemGetInfo (~0.9ms each — ~55ms of
the ~60ms decode step at 61 layers). Remember the request each class
settled at and return immediately while dims and budget are unchanged;
the mark is cleared on release and re-evaluated on any budget change.

3. Pipeline the streamed expert upload path (23.5 -> ~24.5)

A cache miss paid ~2.3ms for a ~13.5 MiB expert: each of the three
tensor copies pread()s the whole tensor into pinned staging as a single
64 MiB-chunk pass and then fully synchronizes the upload stream, so
reads and uploads never overlap. Split the copy into 2 MiB chunks
(DS4_CUDA_STREAM_COPY_CHUNK_MB) through the existing 4-buffer ring
and issue the three tensors of an expert as one async group with a
single flush.

4. Plain pageable uploads when there is no O_DIRECT path (~24.5 -> 29-32)

With the model map backed by page cache/tmpfs, uploading straight from
the (pageable) map runs ~3x faster than the staged pread bounce on this
box (~2.3 -> ~0.8ms per expert). Default to the plain copy only when no
O_DIRECT read path is active — real direct-I/O SSD streaming keeps the
staged path and its page-cache bypass. DS4_CUDA_STREAM_MISS_PLAIN_COPY=1/0
forces either way.

Numbers

RTX PRO 6000 (96 GB), DeepSeek V4 Flash q4 GGUF in tmpfs,
--ssd-streaming --ssd-streaming-cache-experts 7000 --ctx 32768 --prefill-chunk 1024, 200 generated tokens per request, single stream,
steady state (2-3 requests of warm-up):

configuration	tok/s
main (no resident cache)	~2.3 flat
+ resident cache	16.7
+ settled budget	23.5
+ pipelined uploads	~24.5
+ plain pageable uploads	29-32

Cache hit rate at steady state is ~96%; what remains of the step is
roughly one third miss uploads, the rest compute and the per-layer
router readback. Greedy output is unchanged in every A/B above
(cache on vs off, staged vs plain uploads — the copies are byte-exact).

With --batched-session N the resident cache is shared by the
sessions, so the per-session pipelining scales aggregates too: on top
of the #604 branch I measured 30 / 34 / 36.5 tok/s aggregate at 1/2/4
concurrent clients with greedy streams identical to the solo run.

Notes and limitations

• CUDA only: Metal keeps its own (already resident-capable) cache
  paths untouched, and the ROCm layer-cache path is untouched too.
• The cache lives on the single-GPU streaming placement (same
  constraint as the selected staging buffers).
• Exact configs and per-step measurements are in the commit messages;
  I can share the bench harness if useful. If you had a different shape
  in mind for streaming caching, happy to adapt or split this.