Skip to content

v0.23.7

Choose a tag to compare

View all tags
@github-actions github-actions released this 11 Aug 03:35
· 15 commits to master since this release
v0.23.7
253add5
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Highlights

  • Added vLLM hybrid KV cache model support, including aligned FullAttention + Mamba layouts and hybrid MLA handling.
  • Added host-local endpoint shards for cross-host tensor parallel deployments. Each PegaFlow server can now serve the contiguous TP ranks on its host while preserving a shared logical cache namespace.

Compatibility

  • The load RPC changed incompatibly to carry grouped layers and per-group destinations. Upgrade PegaFlow servers and vLLM connectors together.
  • Cross-host endpoint shards currently support equal, contiguous TP-only layouts. Pipeline, decode-context, and prefill-context parallel combinations are rejected.
  • Published wheels cover Python 3.10-3.14 on x86_64 for CUDA 12 and CUDA 13. aarch64 systems currently need a source build.

Validation

  • Qwen3.5-4B hybrid-model correctness E2E: 5 tests passed.
  • Kimi K3 on B300 x 8: cold and warm outputs matched at cached boundaries; measured median speedup was 2.42x.
  • Split local-server E2E: 33.81 req/s cold and 42.74 req/s warm, with matching outputs.

Known issue

  • P2P prefetches for multiple endpoint shards currently start serially within one scheduler poll. This affects latency rather than KV correctness and is tracked in #424.

Full Changelog: v0.23.6...v0.23.7

Assets 12
Loading