Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update kv quantization and inference guide #1412

Merged
merged 19 commits into from
Apr 16, 2024
Merged

Update kv quantization and inference guide #1412

merged 19 commits into from
Apr 16, 2024

Conversation

lvhan028
Copy link
Collaborator

@lvhan028 lvhan028 commented Apr 8, 2024 •
edited

after #1377

@lvhan028 lvhan028 added the documentation Improvements or additions to documentation label Apr 8, 2024
@lvhan028 lvhan028 mentioned this pull request Apr 8, 2024
Closed
This was referenced Apr 10, 2024
Merged
Merged
@zhulinJulia24
Copy link
Collaborator

@zhulinJulia24 react testcase

@lvhan028 lvhan028 mentioned this pull request Apr 11, 2024
Closed
@zhulinJulia24
Copy link
Collaborator

https://github.com/InternLM/lmdeploy/blob/main/docs/zh_cn/supported_models/supported_models.md 这个文档要更新么？

@lvhan028
Copy link
Collaborator Author

https://github.com/InternLM/lmdeploy/blob/main/docs/zh_cn/supported_models/supported_models.md 这个文档要更新么？

要的。多谢多谢，我等下更新

RunningLeon
RunningLeon reviewed Apr 16, 2024
View reviewed changes
notebook.md Outdated Show resolved Hide resolved
@zhulinJulia24
Copy link
Collaborator

zhulinJulia24 commented Apr 16, 2024 •
edited

https://github.com/InternLM/lmdeploy/blob/3e9166d17f1917bdc7799ff4dfe99905e3543f96/docs/zh_cn/quantization/w4a16.md

这个中文的文档也可改internlm -> internlm2

https://github.com/InternLM/lmdeploy/blob/3e9166d17f1917bdc7799ff4dfe99905e3543f96/docs/en/quantization/w4a16.md
模型没改全，还有部分为internlm-chat-7b

image

zhulinJulia24
zhulinJulia24 reviewed Apr 16, 2024
View reviewed changes
docs/en/supported_models/supported_models.md Outdated
| YI | 6B - 34B | Yes | No | No |
| InternVL-Chat | - | Yes | No | No |
| Model | Size | FP16/BF16 | KV INT8 | KV INT4 | W4A16 |
| ------------------ | ---------- | --------- | --------- | --------- | ----- |
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

image

zhulinJulia24
zhulinJulia24 approved these changes Apr 16, 2024
View reviewed changes
Copy link
Collaborator

@zhulinJulia24 zhulinJulia24 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@lvhan028 lvhan028 merged commit 3247e22 into InternLM:main Apr 16, 2024
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@RunningLeon RunningLeon RunningLeon left review comments

@zhyncs zhyncs zhyncs left review comments

@lzhangzz lzhangzz lzhangzz approved these changes

@zhulinJulia24 zhulinJulia24 zhulinJulia24 approved these changes

Assignees
No one assigned
Labels
documentation Improvements or additions to documentation
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
5 participants
@lvhan028 @zhulinJulia24 @lzhangzz @RunningLeon @zhyncs