[FIX] Make flash_attn optional
#3269
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This reverts commit 2daf23a.
WoosukKwon
changed the title
|
Hi @WoosukKwon, why you revert your refraction in #3005? |
zhuohan123
reviewed
Mar 8, 2024
|
I think it would be safe to simply disable the flash attention backend if |
|
@zhuohan123 @AlpinDale Thanks for the suggestion. Just updated the PR accordingly. Please take another look. |
WoosukKwon
changed the title
WoosukKwon
changed the title
flash_attn optional
|
Thank you @WoosukKwon, I tested this on a few environments and this restored the original behavior for me. |
zhuohan123
approved these changes
Mar 8, 2024
|
@zhuohan123 Let's remove it for this fix and bring it back once we find a more stable way to do it. |
This was referenced Mar 8, 2024
AdrianAbeyta
pushed a commit
to ROCm/vllm
that referenced
this pull request
Mar 8, 2024
joerunde
pushed a commit
to IBM/vllm
that referenced
this pull request
Mar 11, 2024
Cherry-picked from vllm-project/vllm#3269
joerunde
pushed a commit
to IBM/vllm
that referenced
this pull request
Mar 11, 2024
Cherry-picked from vllm-project/vllm#3269 Signed-off-by: Joe Runde <Joseph.Runde@ibm.com>
joerunde
pushed a commit
to IBM/vllm
that referenced
this pull request
Mar 11, 2024
joerunde
pushed a commit
to IBM/vllm
that referenced
this pull request
Mar 11, 2024
Signed-off-by: Joe Runde <Joseph.Runde@ibm.com>
joerunde
pushed a commit
to IBM/vllm
that referenced
this pull request
Mar 12, 2024
Signed-off-by: Joe Runde <Joseph.Runde@ibm.com>
dtransposed
pushed a commit
to afeldman-nm/vllm
that referenced
this pull request
Mar 26, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The FlashAttention backend introduced in #3005 causes build errors in some environments and increase the package size significantly (44 MB -> 160 MB) as the vLLM package now includes the
flash-attn(116 MB) package. This PR addresses this by removingflash_attnfrom vLLM's dependency and enables falling back to xFormers whenflash_attnis not found.