Fixes for continuous batching #40828
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
This was referenced Sep 11, 2025
ArthurZucker
approved these changes
Sep 12, 2025
src/transformers/generation/continuous_batching/continuous_api.py
Outdated
Show resolved
Hide resolved
src/transformers/generation/continuous_batching/continuous_api.py
Outdated
Show resolved
Hide resolved
I think there were issues last time, but I can check
Tried that it does not work for a few reasons, will look into restoring asap |
McPatate
reviewed
Sep 12, 2025
| # We centralize the logger here to coordinate between logging and progress bar | ||
| logger = logging.getLogger("ContinuousBatchingLogger") | ||
| logger.setLevel(logging.INFO) | ||
| # logger.setLevel(logging.INFO) |
There was a problem hiding this comment.
Yes, seems like default should not be INFO. I can remove the comment next time, I agree it will be cleaner :)
ErfanBaghaei
pushed a commit
to ErfanBaghaei/transformers
that referenced
this pull request
Sep 25, 2025
* Fix for CB attn mask and refactor * Tests for CB (not all passing) * Passing tests and a logger fix * Fixed the KV metrics that were broken when we moved to hybrid alloc * Fix circular import and style * Added tests for FA * Unfolded test to have device expectations * Fixes for H100 * more fixes for h100 * H100 are good * Style * Adding some comments from huggingface#40831 * Rename test * Avoid 1 letter variables * Dictonnary is only removed during kwargs * Test for supported sample * Fix a unvoluntary slice * Fixes for non-sliced inputs and small example improvments * Slice inputs is more understandabe * Style
vijayabhaskar-ev
pushed a commit
to vijayabhaskar-ev/transformers
that referenced
this pull request
Oct 2, 2025
* Fix for CB attn mask and refactor * Tests for CB (not all passing) * Passing tests and a logger fix * Fixed the KV metrics that were broken when we moved to hybrid alloc * Fix circular import and style * Added tests for FA * Unfolded test to have device expectations * Fixes for H100 * more fixes for h100 * H100 are good * Style * Adding some comments from huggingface#40831 * Rename test * Avoid 1 letter variables * Dictonnary is only removed during kwargs * Test for supported sample * Fix a unvoluntary slice * Fixes for non-sliced inputs and small example improvments * Slice inputs is more understandabe * Style
yuchenxie4645
pushed a commit
to yuchenxie4645/transformers
that referenced
this pull request
Oct 4, 2025
* Fix for CB attn mask and refactor * Tests for CB (not all passing) * Passing tests and a logger fix * Fixed the KV metrics that were broken when we moved to hybrid alloc * Fix circular import and style * Added tests for FA * Unfolded test to have device expectations * Fixes for H100 * more fixes for h100 * H100 are good * Style * Adding some comments from huggingface#40831 * Rename test * Avoid 1 letter variables * Dictonnary is only removed during kwargs * Test for supported sample * Fix a unvoluntary slice * Fixes for non-sliced inputs and small example improvments * Slice inputs is more understandabe * Style
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Some architectures like
llamaalter the attention mask if it is not a tensor, which was not compatible with the way CB created and handled the attention mask. Now, arguments likeattention_mask,cumulative_seqlens_kandmax_seqlen_kare tensors or ints unless the model is hybrid, in which case they are dictonnaries. This is the main fix, but PR also:eager_paged