Skip to content
This repository was archived by the owner on Nov 21, 2024. It is now read-only.

async scheduling for decoding

Latest
Latest

Choose a tag to compare

View all tags
@noooop noooop released this 08 Nov 10:04
· 1 commit to main since this release
v0.3.0
290537b

非常dirty的实现自回归模型（chat models）异步调度

7b-fp8
3b-bf16

两倍的延迟实现10%的峰值吞吐提高
效果有点令人扣头

会在 https://github.com/noooop/wde 库进行进一步优化代码，希望能有性能的进一步提升

祝好
light-vllm v0.3.0

Assets 2
Loading