Add a mode to avoid clone() in DDPSink #122927
Conversation
pritamdamania87
requested review from
albanD,
jbschlosser and
mikaylagawarecki
as code owners
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/122927
Note: Links to docs will display an error until the docs builds have been completed. ✅ No FailuresAs of commit 48e2ef7 with merge base e70bf23 ( This comment was automatically generated by Dr. CI and updates every 15 minutes. |
pytorch-bot
bot
added
the
oncall: distributed
janeyx99
added
the
triaged
|
Thanks for the review @fegin and @rohan-varma! |
|
@pytorchbot merge |
|
Pull workflow has not been scheduled for the PR yet. It could be because author doesn't have permissions to run those or skip-checks keywords were added to PR/commits, aborting merge. Please get/give approval for the workflows and/or remove skip ci decorators before next merge attempt. If you think this is a mistake, please contact PyTorch Dev Infra. |
|
@fegin @rohan-varma Could you approve the appropriate workflows as suggested in #122927 (comment)? Thanks! |
|
@pytorchbot merge |
Merge failedReason: This PR needs a If not, please add the To add a label, you can comment to pytorchbot, for example For more information, see Details for Dev Infra teamRaised by workflow job |
|
@pytorchbot merge |
Merge startedYour change will be merged once all checks pass (ETA 0-4 Hours). Learn more about merging in the wiki. Questions? Feedback? Please reach out to the PyTorch DevX Team |
Merge failedReason: 3 mandatory check(s) failed. The first few are: Dig deeper by viewing the failures on hud |
|
@pytorchbot merge -f "flaky windows test" |
|
You are not authorized to force merges to this repository. Please use the regular |
|
@pytorchbot merge |
Merge startedYour change will be merged once all checks pass (ETA 0-4 Hours). Learn more about merging in the wiki. Questions? Feedback? Please reach out to the PyTorch DevX Team |
Merge failedReason: 3 mandatory check(s) failed. The first few are: Dig deeper by viewing the failures on hud |
|
@pytorchbot rebase |
|
@pytorchbot started a rebase job onto refs/remotes/origin/viable/strict. Check the current status here |
|
Successfully rebased |
pytorchmergebot
force-pushed
the
user/pdamania/ddp_avoid_clone
branch
from
a72a5b8 to
48e2ef7
Compare
|
@pytorchbot merge |
Merge startedYour change will be merged once all checks pass (ETA 0-4 Hours). Learn more about merging in the wiki. Questions? Feedback? Please reach out to the PyTorch DevX Team |
DDPSink clones the outputs of DDP to avoid in-place modification of loss (see pytorch#61982). However, when outputs are really large (2-3GB) this adds a lot of overhead for peak memory. As a result, adding a mode to avoid this clone in cases where users are not modifying loss in-place. Pull Request resolved: pytorch#122927 Approved by: https://github.com/fegin, https://github.com/rohan-varma
DDPSink clones the outputs of DDP to avoid in-place modification of loss (see pytorch#61982). However, when outputs are really large (2-3GB) this adds a lot of overhead for peak memory. As a result, adding a mode to avoid this clone in cases where users are not modifying loss in-place. Pull Request resolved: pytorch#122927 Approved by: https://github.com/fegin, https://github.com/rohan-varma
7 participants
DDPSink clones the outputs of DDP to avoid in-place modification of loss (see #61982). However, when outputs are really large (2-3GB) this adds a lot of overhead for peak memory.
As a result, adding a mode to avoid this clone in cases where users are not modifying loss in-place.
cc @mrshenli @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu @penguinwu @fegin @XilunWu @wanchaol @fduwjj @wz337 @tianyu-l @wconstab @yf225 @chauhang