Highlights
- Cache pressure now prefers reclaiming remote-fetched KV replicas before local/SSD residents, reducing thrashing of primary copies under multi-node load (#412).
- MetaServer insert responses can mark a newly registered replica as reclaimable once a block has three or more live owners, so redundant third-or-later copies are demoted for early eviction (#413).
- Prefill/decode handoff can complete remote prefetch with partial tail blocks: decode loads prefill-saved partial prompt blocks while recomputing the logits token, and
pegaflow.wait_for_full_prefixkeeps the query in prefetch until the remaining prefix is ready (#411). - Added a NIXL P/D deployment example for GLM-5.2 FP8 with MTP and MultiConnector (#410).
Cache replacement policy
Source-based classes (from #412), refined by MetaServer hints (from #413):
| Event | Class behavior |
|---|---|
| Local save / SSD prefetch creates a resident block | retained |
| RDMA/P2P fetch creates a resident block | reclaimable |
| MetaServer insert: new owner and post-insert count ≥ 3 | demote matching retained resident to reclaimable |
| Duplicate insert or cache hit | refresh current class LRU; do not migrate |
| Memory pressure | evict reclaimable LRU, then retained LRU |
reclaimable_hashes is an insert-time signal, not a continuously synchronized replica count. Pega server and MetaServer are expected at the same version for this release.
Prefetch and P/D (#411)
pegaflow.wait_for_full_prefixkeeps a query in PegaFlow's prefetch state until the remaining prefix is complete.- Decode can load prefill-saved partial prompt blocks and recompute the logits token.
- CUDA IPC registration selects and synchronizes the correct device.
- Documented P/D hash seed, hash algorithm, and connector settings.
- Validated on H200 x8 with GLM-5.2 FP8: multi-turn and long-prompt runs showed partial-tail hits and greedy token parity with cold baseline (no TTFT improvement claimed).
Docs
- NIXL P/D deployment example with shared hash/layout invariants and D-to-P reuse validation (#410).
Full Changelog: v0.23.4...v0.23.5