DDPCPU not woking with DDPPlugin(find_unused_parameters=True) in Ver 1.2.0

[rudaoshi]

🐛 Bug

when using accelerator="ddp_cpu" together with plugins=[DDPPlugin(find_unused_parameters=True)] to create a trainer, the trainer will cause the program tries to re-run its self (and recreate the trainer) and finally then failed at checking gpu devices.

Please reproduce using the BoringModel

    trainer = Trainer(
        max_epochs=1,
        gpus=0,
        accelerator="ddp_cpu",
        num_processes=4,
        plugins=[DDPPlugin(find_unused_parameters=True)],
    )

To Reproduce

Expected behavior

Environment

• PyTorch Version (e.g., 1.0): 1.7.1
• OS (e.g., Linux): osX
• How you installed PyTorch (conda, pip, source): pip
• Build command you used (if compiling from source):
• Python version: 3.8.7
• CUDA/cuDNN version: No
• GPU models and configuration: gpus=None
• Any other relevant information:

Additional context

[awaelchli]

Hey! Thanks for reporting. I will get on this asap. I know what the fix is.

Side note: Before 1.2 accelerator="ddp_cpu" would always just use the spawn method, i.e., the "ddp_spawn" equivalent you may know for gpus. However, now in 1.2, and since you pass the DDPPlugin specifically to the plugins list, it forces the normal DDP on CPU. We simply have never tested this case, but I'll look into it now and send a PR :)

[rudaoshi]

Thank you for your response. So is there a DDPCPUPlugin for ddp_cpu accelerator so that I can set the find_unused_parameters = true?

Actually, I'd rather suggest that set the find_unused_parameters default to true. Since the pretrained model is the main stream methodology in deep learning, and there are aways unused parameters in the pretrained model.