Repository navigation
Will training be faster and efficient without Torch.cuda.synchronize() #98
Unanswered
abhinavsb3
asked this question in
Q&A
Replies: 1 comment
|
Yes, in general training will be faster without calling Why:
So you usually do not need it for normal training. Use it only when you specifically need:
So the practical rule is: for regular training: remove it If you call it every microstep just for logging speed, then yes, that can slow training. |
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
When training ,other than knowing the token processing rate related information on GPU do we still need to run torch.cuda.synchronize() in each microstep?since it wait for the GPU to finish work won't it slow the training? Wouldn't it make training faster without synchronization?
All reactions