Multiple fixes for functional collectives. #95897
Conversation
kumpera
requested review from
H-Huang,
albanD,
awgu,
fegin,
kwen2501,
mrshenli,
rohan-varma,
wanchaol and
zhaojuanmao
as code owners
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/95897
Note: Links to docs will display an error until the docs builds have been completed. ❌ 1 FailuresAs of commit 82e9af2: BROKEN TRUNK - The following jobs failed but were present on the merge base 004bcff:👉 Rebase onto the `viable/strict` branch to avoid these failures
This comment was automatically generated by Dr. CI and updates every 15 minutes. |
|
@kumpera has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
1 similar comment
|
@kumpera has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
|
oops @wanchao already landed reduce_scatter, so that op also needs the derivatives.yml and codegen fix. you can do it in a separate PR if its easier to land this first |
kumpera
added
topic: bug fixes
|
@pytorchmergebot merge |
Merge startedYour change will be merged once all checks pass (ETA 0-4 Hours). Learn more about merging in the wiki. Questions? Feedback? Please reach out to the PyTorch DevX Team |
Merge failedReason: 43 mandatory check(s) failed. The first few are:
Dig deeper by viewing the failures on hud |
_functional_collectives.py: Ensure we always wait all collectives. derivatives.yaml: mark all_reduce as non differentiable gen_variable_type.py: Add all_reduce to DONT_ENFORCE_TENSOR_IMPL_USE_COUNT common_dtensor.py: replace dist.barrier with all_reduce
|
@kumpera has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
|
@pytorchmergebot merge -f "the inductor failure is unrelated" |
Merge startedYour change will be merged immediately since you used the force (-f) flag, bypassing any CI checks (ETA: 1-5 minutes). Learn more about merging in the wiki. Questions? Feedback? Please reach out to the PyTorch DevX Team |
_functional_collectives.py: Ensure we always wait all collectives. derivatives.yaml: mark all_reduce as non differentiable gen_variable_type.py: Add all_reduce to DONT_ENFORCE_TENSOR_IMPL_USE_COUNT common_dtensor.py: replace dist.barrier with all_reduce Pull Request resolved: pytorch/pytorch#95897 Approved by: https://github.com/wconstab, https://github.com/fegin
_functional_collectives.py: Ensure we always wait all collectives. derivatives.yaml: mark all_reduce as non differentiable gen_variable_type.py: Add all_reduce to DONT_ENFORCE_TENSOR_IMPL_USE_COUNT common_dtensor.py: replace dist.barrier with all_reduce Pull Request resolved: pytorch/pytorch#95897 Approved by: https://github.com/wconstab, https://github.com/fegin
_functional_collectives.py: Ensure we always wait all collectives. derivatives.yaml: mark all_reduce as non differentiable gen_variable_type.py: Add all_reduce to DONT_ENFORCE_TENSOR_IMPL_USE_COUNT common_dtensor.py: replace dist.barrier with all_reduce Pull Request resolved: pytorch#95897 Approved by: https://github.com/wconstab, https://github.com/fegin
…arts of the codebase (#96460) Recent master breakage on focal and bionic PTD tests since we switched to all_reduce in #95897 Pull Request resolved: #96460 Approved by: https://github.com/fegin
5 participants
_functional_collectives.py: Ensure we always wait all collectives.
derivatives.yaml: mark all_reduce as non differentiable
gen_variable_type.py: Add all_reduce to DONT_ENFORCE_TENSOR_IMPL_USE_COUNT
common_dtensor.py: replace dist.barrier with all_reduce