-
Notifications
You must be signed in to change notification settings - Fork 243
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support torch deepseek moe #1163
Conversation
b5c5c45
to
2079611
Compare
2079611
to
53944ab
Compare
CUDA_VISIBLE_DEVICES=6,7 lmdeploy chat torch /nvme/shared_data/deepseek/deepseek-moe-16b-chat/ --tp 2 raises
Do we need to install flash-attn? |
@lvhan028 yes I guess. |
新模型的支持,PR需要测试以下功能:
|
All test passed. |
lmdeploy/model.py
Outdated
ret = f'{self._bos}' | ||
for _, msg in enumerate(messages): | ||
role, content = msg['role'], msg['content'] | ||
if role == self.user: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
no attribute user
. It is _user
for _, msg in enumerate(messages): | ||
role, content = msg['role'], msg['content'] | ||
if role == self.user: | ||
ret += f'{self._user} {content}\n\n{self._assistant}' |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
Add deepseek moe support
require
document would be updated after #1075 merged.