Skip to content

Add support for LLaMA-2 - #505

Merged
zhuohan123 merged 9 commits into
mainfrom
support-llama-2
Jul 20, 2023
Merged

Add support for LLaMA-2#505
zhuohan123 merged 9 commits into
mainfrom
support-llama-2

Conversation

@zhuohan123

@zhuohan123 zhuohan123 commented Jul 18, 2023 •
edited
Loading

Copy link
Copy Markdown
Member

Fix #501

Update: this PR has some correctness issues on 70B models. Will look into it.

@WoosukKwon This PR is ready to go. Please review and let's merge it!

@zhuohan123 zhuohan123 changed the title [WIP] Add support for LLaMA-2 Add support for LLaMA-2 Jul 18, 2023
@gesanqiu

gesanqiu commented Jul 19, 2023

Copy link
Copy Markdown
Contributor

I notice that you didn't new a llama2.py, so this PR compatible with both llama and llama2?

@zhuohan123

zhuohan123 commented Jul 19, 2023

Copy link
Copy Markdown
Member Author

I notice that you didn't new a llama2.py, so this PR compatible with both llama and llama2?

Yes, old model should still be compatible.

@LiuXiaoxuanPKU LiuXiaoxuanPKU mentioned this pull request Jul 20, 2023
Closed
@zhuohan123
zhuohan123 requested a review from WoosukKwon July 20, 2023 05:54
WoosukKwon
WoosukKwon approved these changes Jul 20, 2023
View reviewed changes

@WoosukKwon WoosukKwon left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@zhuohan123 Awesome! Thanks for the great work! Just left minor comments.

Please double-check that this PR doesn't break LLaMA V1 and other models using RoPE, before merging the PR.

Comment thread csrc/pos_encoding_kernels.cu
Comment thread vllm/model_executor/models/llama.py Outdated
Comment thread vllm/model_executor/models/llama.py Outdated
Comment thread vllm/model_executor/models/llama.py Outdated
@zhuohan123
zhuohan123 merged commit 6fc2a38 into main Jul 20, 2023
gqjia added a commit to gqjia/vllm that referenced this pull request Jul 21, 2023
@gqjia
Merge pull request #1 from vllm-project/main
c33a2f4 
Add support for LLaMA-2 (vllm-project#505)
@void-main void-main mentioned this pull request Jul 21, 2023
Open
@zhuohan123
zhuohan123 deleted the support-llama-2 branch July 25, 2023 21:59
@ri938

ri938 commented Jul 27, 2023

Copy link
Copy Markdown
Contributor

I am getting an error when trying to load some LLama V1 models:

LlamaConfig object has no attribute 'num_key_value_heads'

@HarrisonBT

HarrisonBT commented Jul 28, 2023

Copy link
Copy Markdown

WARNING 07-28 03:23:18 scheduler.py:196] Input prompt (2716 tokens) is too long and exceeds limit of 4096

@tuyaao

tuyaao commented Jul 31, 2023

Copy link
Copy Markdown

I am getting an error when trying to load some LLama V1 models:

LlamaConfig object has no attribute 'num_key_value_heads'
same error from me on lastest master commit: 953f28c

@pseudotensor pseudotensor mentioned this pull request Aug 28, 2023
Open
hongxiayang pushed a commit to hongxiayang/vllm that referenced this pull request Feb 13, 2024
@zhuohan123
Add support for LLaMA-2 (vllm-project#505)
9c84946
pi314ever pushed a commit to pi314ever/vllm that referenced this pull request Nov 20, 2024
@kzawora-intel @michalkuligowski
Terminate ray workers on ray_hpu_executor shutdown (vllm-project#505)
96467d8 
Co-authored-by: Michał Kuligowski <mkuligowski@habana.ai>
amy-why-3459 pushed a commit to amy-why-3459/vllm that referenced this pull request Sep 15, 2025
@wangxiyuan
[V1] clean up V1 code (vllm-project#505)
c7f6584 
Clean up V1 code:
1. remove useless code.
2. format code to be clear.

Signed-off-by: wangxiyuan <wangxiyuan1007@gmail.com>
cursor Bot pushed a commit to Shirley125/vllm_epd that referenced this pull request Jan 22, 2026
@gcanlin
[Bugfix][NPU] Add _model_forward for ModelRunner (vllm-project#505)
be81443 
Signed-off-by: gcanlin <canlinguosdu@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@WoosukKwon WoosukKwon WoosukKwon approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Support LLaMA-2

6 participants

@zhuohan123 @gesanqiu @ri938 @HarrisonBT @tuyaao @WoosukKwon