Repository navigation
vllm-plugin-FL v0.3.0
Part of FlagOS 2.2.
Compatibility line: vLLM 0.24.0.
Changes since v0.2.0 (FlagOS 2.1).
64 pull requests and 0 additional commits.
Source: v0.3.0, commit f1052770d3ff.
New Features
- Add weak_ref_tensor C++ extension build system with CMake (#231) by @creaspoggg
- feat(metax): adapt MetaX C550 backend for vLLM 0.20.2 (#241) by @physics31415926
- feat: upgrade flagcx connector for support vllm0.20-0.24(cherry-pick from main branch) (#295) by @leoda1
- feat(quantization): adapt W8A8 inference to vLLM 0.24 (#336) by @rdzhu225
- feat(compilation): add BreakableCUDAGraphWrapper support for OOT vendor backends (vLLM 0.24.0+) (#342) by @physics31415926
- feat(vllm-0.24): add Qwen3.5 text-only runtime compatibility (#383) by @CherryLemon
- feat(ascend): port E2E-verified ascend support from 0.20.2 to v0.3.0-dev (vllm 0.24.0) (#387) by @tengqm
- feat(sunrise): port attention backend to vLLM 0.24.0 (CUSTOM registration + ptpu memory_stats shim) (#391) by @tengqm
- feat(kunlunxin): port Kunlunxin vendor backend to vLLM 0.24.0 (#401) by @tengqm
- feat(flagcx connector): Prometheus KV-transfer metrics + port #315 from release/0.2 (#418) by @leoda1
- feat: support custom throughput test cases (#426) by @cyber-pioneer
- feat(cambricon): port MLU590 + neuware4.4.3 compat to main (#431) by @tengqm
- feat(enflame): port GCU300 enablement bundle to main (#432) by @tengqm
- feat(dispatch): support appending FlagGems blacklist entries (#439) by @CherryLemon
- add vllm-plugin-fl for txda based on empty vllm==0.24.0. (#447) by @tsingmicro-public-e
- feat(tsingmicro): enable vllm 0.24.0 on txda (complement #447) (#460) by @tengqm
Performance
- perf(nvidia): add opt-in Hopper long-context routing (#384) by @CherryLemon
- perf(vllm-0.24): batch MTP xGrammar masks (#414) by @ceci3
Hardware Support
- MUSA support update to vllm-0.20.2 (#176) by @luoyc123
- Arm: integrate Qwen packed W4A8 and GDN on CPU (#433) by @kevinzs2048
Improvements
- refactor(musa): remove direct flaggems calls from vendor backend (#376) by @physics31415926
Documentation
- docs: update README.md for vLLM v0.24.0 (#393) by @physics31415926
- [DOC] Add version doc (#403) by @cyber-pioneer
CI/Infrastructure
- Migrate CI related code from release/0.2 to main (#415) by @xmhubj
- Enable Hygon workflow for vLLM 0.24.0 (#436) by @xmhubj
- ci: add /rerun-failed-ci and /cancel-ci PR comment commands (#480) by @lentil1016
Bug Fixes
- Fix CachedOp fast-path diagnostics and fallback behavior (#206) by @li199959
- Fix gems info file format (#220) by @cyber-pioneer
- Fix gems DeviceDetector api (#236) by @cyber-pioneer
- fix(moe): use valid_grouping closure in GroupedTopKRouterFL (#237) by @ZenithJoyH
- fix: make mcoplib._C pre-load unconditional in register_op_schemas (#250) by @physics31415926
- fix hy3-previw Accuracy problem in fusedmoeFL (#251) by @ZenithJoyH
- Fix is_monolithic missing when setting VLLM_FL_PREFER_ENABLED=0 (#252) by @cyber-pioneer
- fix bug for hygon import DeviceDetector (#260) by @creaspoggg
- fix mtp in 0.24.0 (#334) by @ceci3
- fix mtp draft_token (#337) by @ceci3
- fix: support official vLLM 0.24 CUDA stable-ABI wheels (#338) by @CherryLemon
- Fix CUDA graph disabling for non-DeepEP backends (#346) by @CherryLemon
- fix(metax): pass group=device_group in compile-friendly all_reduce (#348) by @physics31415926
- fix(metax): consolidate vLLM 0.20.2 compat shims into the plugin (#377) by @tengqm
- fix(worker): preserve DP GPU offset for independent engines (#380) by @CherryLemon
- fix(worker): account for CUDA graph memory in KV cache sizing (#381) by @CherryLemon
- fix(gdn): keep packed decode beta in fp32 (#385) by @CherryLemon
- fix(worker): guard kernel_warmup against missing torchvision on OOT runtimes (#386) by @tengqm
- fix(iluvatar): consolidate triton patches at module level, remove dead code (#406) by @physics31415926
- fix(dispatch): keep available reference fallbacks registered (#440) by @CherryLemon
- fix(quantization): inherit native MXFP8 candidates for OOT (#441) by @CherryLemon
- fix(thead): restore static graph support from #279 (#472) by @physics31415926
- fix(attention): restore FlagGems KV cache updates from #382 (#474) by @physics31415926
- fix(platform): stub torch.distributed._symmetric_memory for torch < 2.8 (#491) by @tengqm
- fix(musa): bind TP workers before runtime patches (#537) by @lentil1016
- fix(platform): stub _symmetric_memory when the package cannot be imported (#549) by @tengqm
Other
- Support empty mode of vllm in PPU (#190) by @cyber-pioneer
- Update gems blacklist for nvidia (#222) by @cyber-pioneer
- Update supoported vendor list (#228) by @cyber-pioneer
- Update fused_experts_impl to gems (#230) by @cyber-pioneer
- support cuda_graph on iluvatar (#232) by @luoyc123
- cherry pick (#136): Ascend refacter fused module. (#240) by @cyber-pioneer
- update gems commit (#243) by @cyber-pioneer
- upgrade: vllm 0.20.2 -> 0.24.0 (#274) by @physics31415926
- adapt(metax): MetaX C550 backend adaptation for vLLM 0.24.0 (#294) by @physics31415926
- adapt(musa): MTT S5000 backend adaptation for vLLM 0.24.0 (#308) by @physics31415926
- adapt(iluvatar): BI-V150 backend adaptation for vLLM 0.24.0 (#310) by @physics31415926
- Cherry-pick PR #297 to v0.3.0-dev: Add thead vendor attention backend (#360) by @physics31415926
Contributors
Thanks to the contributors to the pull requests included in this release.
- @physics31415926 (14 PRs)
- @cyber-pioneer (11 PRs)
- @CherryLemon (10 PRs)
- @tengqm (10 PRs)
- @ceci3 (3 PRs)
- @ZenithJoyH (2 PRs)
- @creaspoggg (2 PRs)
- @lentil1016 (2 PRs)
- @leoda1 (2 PRs)
- @luoyc123 (2 PRs)
- @xmhubj (2 PRs)
- @kevinzs2048
- @li199959
- @rdzhu225
- @tsingmicro-public-e