How to train with multiple GPUs

[xiaohei1001]

Thanks for the great work! Now I have some issues, when I use multiple GPUs (RTX 3090) for training, the GPU memory overflows, while with a single GPU everything works fine. Please tell me how I can use multiple GPUs for training.

[SeanLee97]

hi, @xiaohei1001. Thanks for your suggestion. We will upgrade our code later to support multiple gpus. You can also modify our code to support it by following instructions of transformers trainer https://huggingface.co/docs/transformers/perf_train_gpu_many.