Skip to content

Do not pin DDP device_ids for CPU reference models - #8399

Merged
delock merged 2 commits into
deepspeedai:masterfrom
delock:pr-c-ddp-cpu-reference
Sep 3, 2026
Merged

Do not pin DDP device_ids for CPU reference models#8399
delock merged 2 commits into
deepspeedai:masterfrom
delock:pr-c-ddp-cpu-reference

Conversation

@delock

@delock delock commented Sep 3, 2026

Copy link
Copy Markdown
Collaborator

Problem

tests/unit/v1/zero/test_zero_user_backward.py builds torch DDP reference models (the known-good baseline that ZeRO results are compared against) at five sites:

model_ddp = DDP(model_ddp, device_ids=[rank], output_device=rank)

device_ids=[rank] assumes rank ↔ GPU index. torch's DDP contract only allows device_ids/output_device for single-device GPU modules; CPU modules live on one shared device and must omit them, so multi-rank CPU runs died inside the DDP constructor with:

ValueError: DistributedDataParallel device_ids and output_device arguments only work with single-device/multiple-device GPU modules or CPU modules, ...

Change

Route all five sites through one helper:

def wrap_ddp_reference(model, device, rank):
    # Only indexed devices take device_ids/output_device; CPU modules live on one shared device.
    if torch.device(device).type == 'cpu':
        return DDP(model)
    return DDP(model, device_ids=[rank], output_device=rank)

Design notes:

  • the condition restates the exact precondition torch's own DDP constructor enforces, in torch's device vocabulary — it follows the model's actual device rather than the global accelerator configuration;
  • a single helper means new reference-model sites cannot forget the branch (the first fix round in [DON'T MERGE] Run multi-rank CPU unit tests in CI via LOCAL_SIZE #8381 missed 4 of the 5 sites for exactly this reason);
  • GPU behavior is unchanged.

Validation

Validated as part of the multi-rank CPU CI experiment in #8381: all DDP-constructor failures were eliminated (the file's few remaining failures there are unrelated — see the triage table in that PR), zero regressions vs the same-commit baseline.

test_zero_user_backward builds torch DDP reference models at five
sites via DDP(model, device_ids=[rank], output_device=rank). torch's
DDP contract only allows device_ids/output_device for single-device
GPU modules; CPU modules live on one shared device and must omit them,
so multi-rank CPU runs failed inside the DDP constructor.

Route all five sites through one wrap_ddp_reference helper that checks
the model's torch device type - the same precondition torch's own DDP
constructor enforces - and pins device ids only for indexed devices.
GPU behavior is unchanged.

Validated as part of the multi-rank CPU experiment in deepspeedai#8381: all
DDP-constructor failures were eliminated (the file's few remaining
failures there are unrelated, see its triage table); zero regressions
vs the same-commit baseline.

Signed-off-by: Guokai Ma <guokai.ma@intel.com>

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 41e1dc720b

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

return model, optimizer


def wrap_ddp_reference(model, device, rank):

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Add the required sign-off trailer

This is a non-merge commit, but its message has no Signed-off-by trailer, so it does not satisfy the repository's commit requirements and may be rejected by the DCO/CI checks. Recreate the commit with git commit --signoff using the configured Git identity.

AGENTS.md reference: AGENTS.md:L8-L8

Useful? React with 👍 / 👎.

@delock
delock enabled auto-merge September 3, 2026 09:54
@delock
delock added this pull request to the merge queue Sep 3, 2026
Merged via the queue into deepspeedai:master with commit d3a1b68 Sep 3, 2026
13 checks passed
@delock
delock deleted the pr-c-ddp-cpu-reference branch September 3, 2026 11:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants