Skip to content

v1.2.0

Latest
Latest

Choose a tag to compare

View all tags
@shihaobai shihaobai released this 04 Aug 12:27
· 14 commits to main since this release
v1.2.0
65c174e
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

This release introduces RL serving with verl, disaggregated ViT serving, a hybrid-cache architecture for linear-attention models, and a multi-level cache system with cache-aware PD. It also delivers broad model-performance improvements—including expanded MTP speculative decoding and optimizations across prefill, attention, MoE, quantization, and distributed inference—together with extensive stability and correctness fixes across scheduling, caching, multimodal processing, APIs, and long-running services.

Highlights

Reinforcement Learning Serving with verl

LightLLM can now serve as the rollout and inference backend for verl-based reinforcement learning workflows. A dedicated RL control plane supports online model-weight updates through distributed communication, tensors, or CUDA IPC, together with cache flushing, request aborts, generation pause/resume, and GPU-memory release/resume. The integration also includes routing-data capture for MoE training. #1298

Disaggregated ViT Serving

Vision encoding can now run as an independent visual_only service on dedicated GPU nodes. LLM servers use a proxy mode to discover registered visual workers through the config server, dispatch image encoding remotely, and retrieve the resulting embeddings through a shared AFS/Redis-backed cache. This separates ViT compute from language-model inference so the two tiers can be deployed and scaled independently. The release also adds configurable ViT attention backends and independent controls for visual and audio modules. #1234 #1191 #1229

Hybrid Radix Cache with Large and Small Pages

A new dual-granularity Hybrid Radix Cache is introduced for hybrid full-attention and linear-attention models such as Qwen3.5 and Qwen3-Next:

  • Large pages represent multi-block, reusable prefix checkpoints and keep the full-attention KV data together with the corresponding linear-attention convolution and SSM states.
  • Small pages preserve fine-grained or fragmented prefix tails, avoiding coarse page boundaries from limiting cache reuse.
  • Small-page entries can be consolidated into large-page checkpoints and integrated with CPU cache/offload, allowing the cache to retain fine-grained matching while managing recurrent states efficiently.

The design addresses the different storage granularities required by token-level KV cache and recurrent linear-attention state, enabling effective prefix caching for models that combine full and linear attention. #1265 #1422

Multi-Level Cache and Cache-Aware PD

The cache hierarchy now spans GPU, CPU, and disk, with support for quantized CPU KV cache, FP8/INT8 KV formats, NUMA-aware CPU page placement, and faster cache initialization. PD disaggregation gains a cache-aware scheduling path, upgraded NIXL KV transfer, Qwen3.5 support, and stronger lifecycle, health-check, disconnect, and multimodal handling. #997 #1098 #1379 #1399 #1412

What's Changed

New Contributors

Full Changelog: v1.1.0...v1.2.0

Contributors

kingder, blueswhen, and 18 other contributors
Assets 2
Loading