Doesn't run on CPUs when `accelerator="ddp_cpu"` on TPUs

[kaushikb11]

🐛 Bug

Doesn't run on CPUs when accelerator="ddp_cpu" is selected for Trainer on TPUs

To Reproduce

    model = BoringModel()
    trainer = Trainer(
        accelerator="ddp_cpu",
        tpu_cores=8,
        max_epochs=5,
    )

[kaushikb11]

@awaelchli An update on this issue, ddp_cpu works on TPUs when num_processes is default. But I pass values for it (2, 3, etc), it throws an Error.

https://symbolize.stripped_domain/r/?trace=7f8c85b0ff24,7f8c85a4520f,6c65636300316c64&map=
*** SIGTERM received by PID 120425 (TID 120425) on cpu 7 from PID 118970; stack trace: ***
PC: @     0x7f8c85b0ff24  (unknown)  open
    @     0x7f8a124b21e0        976  (unknown)
    @     0x7f8c85a45210  (unknown)  (unknown)
    @ 0x6c65636300316c65  (unknown)  (unknown)
https://symbolize.stripped_domain/r/?trace=7f8c85b0ff24,7f8a124b21df,7f8c85a4520f,6c65636300316c64&map=ca1b7ab241ee28147b3d590cadb5dc1b:7f8a057b3000-7f8a127e5b20
E0707 12:33:03.001385  120425 coredump_hook.cc:250] RAW: Remote crash gathering disabled for SIGTERM.
E0707 12:33:03.015446  120425 process_state.cc:771] RAW: Raising signal 15 with default behavior

[awaelchli]

Is there any known difference of a CPU inside a TPU machine? make sure to omit tpu_cores when running this plugin. We should probably make this an error.

So you are saying

    model = BoringModel()
    trainer = Trainer(
        accelerator="ddp_cpu",
        num_processes=2
        max_epochs=5,
    )

does not work on a TPU machine, but on a regular CPU machine all is good right?

[kaushikb11]

It fails even after omitting the tpu_cores flag.

    model = BoringModel()
    trainer = Trainer(
        default_root_dir=os.getcwd(),
        limit_train_batches=30,
        limit_val_batches=30,
        accelerator="ddp_cpu",
        # tpu_cores=8,
        num_processes=2,
        num_sanity_val_steps=0,
        max_epochs=3,
        weights_summary=None,
    )

Yes, it works as expected on my CPU machine. No worries, I will dig more into it.

[kaushikb11]

Update on this, I commented the XLA interactions in the Lightning code, but still, it was raising issues. Seems like this would be an issue on the tpuvm side if you are running ddp_cpu directly on TPU host machines.