Do not pin DDP device_ids for CPU reference models - #8399
Conversation
test_zero_user_backward builds torch DDP reference models at five sites via DDP(model, device_ids=[rank], output_device=rank). torch's DDP contract only allows device_ids/output_device for single-device GPU modules; CPU modules live on one shared device and must omit them, so multi-rank CPU runs failed inside the DDP constructor. Route all five sites through one wrap_ddp_reference helper that checks the model's torch device type - the same precondition torch's own DDP constructor enforces - and pins device ids only for indexed devices. GPU behavior is unchanged. Validated as part of the multi-rank CPU experiment in deepspeedai#8381: all DDP-constructor failures were eliminated (the file's few remaining failures there are unrelated, see its triage table); zero regressions vs the same-commit baseline. Signed-off-by: Guokai Ma <guokai.ma@intel.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 41e1dc720b
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| return model, optimizer | ||
|
|
||
|
|
||
| def wrap_ddp_reference(model, device, rank): |
There was a problem hiding this comment.
Add the required sign-off trailer
This is a non-merge commit, but its message has no Signed-off-by trailer, so it does not satisfy the repository's commit requirements and may be rejected by the DCO/CI checks. Recreate the commit with git commit --signoff using the configured Git identity.
AGENTS.md reference: AGENTS.md:L8-L8
Useful? React with 👍 / 👎.
Problem
tests/unit/v1/zero/test_zero_user_backward.pybuilds torch DDP reference models (the known-good baseline that ZeRO results are compared against) at five sites:device_ids=[rank]assumes rank ↔ GPU index. torch's DDP contract only allowsdevice_ids/output_devicefor single-device GPU modules; CPU modules live on one shared device and must omit them, so multi-rank CPU runs died inside the DDP constructor with:Change
Route all five sites through one helper:
Design notes:
Validation
Validated as part of the multi-rank CPU CI experiment in #8381: all DDP-constructor failures were eliminated (the file's few remaining failures there are unrelated — see the triage table in that PR), zero regressions vs the same-commit baseline.