-
Notifications
You must be signed in to change notification settings - Fork 4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
DeepSpeed GPU 0 parameter server OOM with 8xV100 #154
Comments
A couple of issues here:
Actually, after my above comments, I just noticed that your comparison is 2xP100 vs 8xV100, and it is unlikely that 2xP100 has more aggregated RAM than 8xV100. This is indeed strange. But please address the questions above to help us investigate this issue. Thanks! |
Sure thing @tjruwase I'm glad to help. I feel I have to explain better the situation:
Since To be more specific about what happened during the training, by using When I do the same thing on 8xV100 I see 8 processes on This led me to think that As always, thank you for your time and help. It's much appreciated! Cheers, |
Hello @tjruwase Thank you and have a nice weekend! Cheers, |
|
@CalogeroZarbo I have some cycles to look into this. Can you please share your deepspeed port of reformer with me? Thanks! |
@tjruwase Sure! It's all published on GitHub: https://github.com/CalogeroZarbo/bioshield Let me know if I can help. Cheers, |
@CalogeroZarbo Thanks so much. That is great! |
@CalogeroZarbo I found the source of problem of GPU0 using more memory than other GPUs. It is because this line sets device to current device, i.e. cuda:0, since device ordinal is not specified. And so all the subsequent .to(device) calls, such as this, were allocating GPU0 memory regardless of the rank. In reality, with deepspeed, you don't need to manipulate device in this way, as deepspeed.initialize() will move the model and parameters to the correct device. And so the fix is to remove all the definition and uses of device. Now, on 4xV100-16GB I get the following Can you please verify the fix, so we can close this issue? Thanks! |
Hi @tjruwase! Thank you for your help! This is a great fix! Sorry to bother you, it was difficult for me to understand that the problem was on my side. Cheers, |
* Squash stage3 v1 (#146) Co-authored-by: Samyam <samyamr@microsoft.com> Co-authored-by: Jeff Rasley <jerasley@microsoft.com> Co-authored-by: Samyam Rajbhandari <samyamr@microsoft.com> Co-authored-by: Olatunji Ruwase <olruwase@microsoft.com> Co-authored-by: Shaden Smith <Shaden.Smith@microsoft.com> Co-authored-by: Shaden Smith <ShadenTSmith@gmail.com> Co-authored-by: eltonzheng <eltonz@microsoft.com> * Fix correctness bug (#147) * formatting fix (#150) * stage3 bugfix (API) update and simplified FP16 Z3 tests (#151) * fp16 Z3 API update and bugfix * revert debug change * ZeRO-3 detach and race condition bugfixes (#149) * trying out ZeRO-3 race condition fix * CUDA sync instead of stream * reduction stream sync * remove commented code * Fix optimizer state_dict KeyError (#148) Co-authored-by: Jeff Rasley <jerasley@microsoft.com> * fix for smaller SGS sizes, ensures each grad is backed by unique tensors (#152) * Simplifying the logic for getting averaged gradients (#153) * skip for now * Z3 Docs redux (#154) * removing some TODOs and commented code (#155) * New Z3 defaults (#156) Co-authored-by: Jeff Rasley <jerasley@microsoft.com> * formatting * megatron external params Co-authored-by: Jeff Rasley <jerasley@microsoft.com> Co-authored-by: Olatunji Ruwase <olruwase@microsoft.com> Co-authored-by: Shaden Smith <Shaden.Smith@microsoft.com> Co-authored-by: Shaden Smith <ShadenTSmith@gmail.com> Co-authored-by: eltonzheng <eltonz@microsoft.com>
Good morning,
I'm opening this issue because I have a small doubt.
I trained my 24 layers deep Reformer, with 768 nodes per layer, using DeepSpeed on 2xP100 successfully.
I tried the same setup on 8xV100 and the GPU 0 crashes due to OOM.
My understanding is that the GPU 0 needs to handle all the results computed also by the other GPUs (acting as parameter-server, I guess) and this lead the GPU 0 to have in memory, in addition to the architecture and the batches to compute, also the parameters data to aggregate.
Is that correct? Is it possible to mitigate this effect somehow?
The only way I found to make 8xV100 to work was to reduce the nodes from 768 to 512, which is counterintuitive since with 8xV100 I should be able to train the same architecture as per 2xP100 but four times faster.
Can you please put some light on this?
Thank you very much in advance for your time!
Cal
The text was updated successfully, but these errors were encountered: