-
Notifications
You must be signed in to change notification settings - Fork 778
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Questions about multi-gpu training #14
Comments
To enable multi-GPU training, you will need to change this line to MultiGPUTrainer. |
@RyanHTR Hello, RyanHTR, can you train the network successfully on multi-GPU? |
@RyanHTR I changed this line to MultiGPUTrainer. But I got an error "TypeError: 'NoneType' object is not callable" which I can't figure it out. Do you have this problem? |
@1900zyh This is not bug. Loss should be None for multi-GPU training. |
@JiahuiYu I think it should be |
@1900zyh Ohhhh I see. Thank you! |
I have 4 GTX 1080Ti GPUs and each gpu can handle batch size of 16 that means if I use all the gpus I can change batch size to 64. But when I do that my GPUs ran out of memory. Because of that Issue I can only train on batch size of 16, whether I use 4 gpus or 1 gpu. |
@bis-carbon The batch size here is the per-gpu batch size. |
Thank you for your quick response and great work. |
@1900zyh @bis-carbon @lipanpeng Hi. Have you figured out the issues that how to use multi gpu for training. If so, kinldy let me know, I am struggling. Thanks in advance |
It's a great work. Does this code support multi-gpu training? I've tried to alter NUM_GPUS and GPU_ID, but it seems like that the code just selects one gpu for training. Is there any clue about it? Thanks.
The text was updated successfully, but these errors were encountered: