Implement a batch_size parameter in the pipeline object

[xegulon]

🚀 Feature request

Implement a batch_size parameter in the pipeline object, so that when we call it, it computes the predictions by batches of sentences and then does get CUDA Out of Memory errors.

Ideally, this optional argument would have a good default, computed from the tokenizer's parameters and the hardware the code is running on.

References to this need in the forum:
https://discuss.huggingface.co/t/how-to-make-pipeline-automatically-scale/7432/3
https://discuss.huggingface.co/t/how-to-change-the-batch-size-in-a-pipeline/8738

Motivation

When making inference on very long list of sentences using the pipeline object, I often get CUDA OOM errors.

Your contribution

I could try :)

[xegulon]

@sgugger ?

[LysandreJik]

Hello @xegulon, this is in line with some work currently underway by @Narsil

[Narsil]

@xegulon,

Batching on inference is something to be very cautious about, because alignment might heavily penalize the speed of inference.

See #11251 and https://gist.github.com/Narsil/ee5c09875e74fa6f018dc6d014f6c06c for more information.

Cuda OOM errors are most likely due to the fact that you are padding way too much, and actually showcase the slow down.

The big refactor mentionned by @LysandreJik is ready here https://github.com/Narsil/transformers/tree/iterable_pipelines

With said PR, you should be able to actually stream all your data to the GPU leading to a massive speedup (like DataLoader), and if you want to do batching because you know it will speedup (please measure real payloads, it's unlikely to be significant, so make sure it is a speedup) you can do it by manually using Dataloader, preprocess, forward and postprocess.
The proposed PR will use DataLoader (for pt) by default if you send lists too. You can also send directly Datasets.

[xegulon]

Great (useful) work @Narsil thanks a lot. Is it planned to be released in v4.10.0?

[Narsil]

I don't think it will make it in time, it's a pretty massive change, we're pulling in stuff bit by bit to make sure we're not breaking anything (we're in a phase where we're strengthening the tests first)

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[Narsil]

@xegulon the modifications have landed in master, can you confirm it speeds up inference without the need for batch_size ?

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.