Skip to content

[CB] Refactor logic for inputs and outputs outside of the main API#43569

Merged
remi-or merged 7 commits intomainfrom
cb-async
Feb 2, 2026
Merged

[CB] Refactor logic for inputs and outputs outside of the main API#43569
remi-or merged 7 commits intomainfrom
cb-async

Conversation

@remi-or
Copy link
Collaborator

@remi-or remi-or commented Jan 28, 2026

This PR refactors all of the logic linked to the inputs and outputs of continuous batching into a new class ContinuousBatchingIOs itself in a new file. This will be quite useful when we introduce async scheduling, because there will be two sets of host-side IOs and one set of device-side IOs.
To avoid having this major refactoring AND the async logic in the same PR, I think it would be good if we can first merge this part, which does not change much, and then add the async logic.
This PR also makes the test file ty compatible and adds a bit more documentation.

@remi-or remi-or requested a review from ArthurZucker January 28, 2026 17:28
@remi-or remi-or self-assigned this Jan 28, 2026
@HuggingFaceDocBuilderDev

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Copy link
Collaborator

@ArthurZucker ArthurZucker left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this makes the code a bit more stateful, but it might be better. I am mostly thinking of multi-gpu runs, this should make stuff simpler.

@remi-or remi-or merged commit e5c8b0d into main Feb 2, 2026
26 checks passed
@remi-or remi-or deleted the cb-async branch February 2, 2026 12:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants