New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

支持vllm? #8

Closed

lx0126z opened this issue Dec 29, 2023 · 2 comments

lx0126z commented Dec 29, 2023

如题，会支持vllm吗？会有更快的推理速度。
https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/models/yi.py

我尝试参照此改写，发现有num_kv_heads 和 share_kv_heads_num 的差异，我尝试改写加载参数时会报错。

他官方的添加方式是这个，https://docs.vllm.ai/en/latest/models/adding_model.html

Owner

wenge-research commented Jan 5, 2024

YAYI和YAYI2都已成功适配vllm加速，我们不久会把代码提交到vllm仓库。

wenge-research closed this as completed

Author

lx0126z commented Jan 8, 2024

暂未在vllm的Pull requests中找到，期待您的提交

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment