multi-gpu training pickle error

[lx709]

Hello guys, thanks for sharing the code of your wonderful work. I was trying to train latent diffusion on my own datasets. When I use a single GPU, everything is ok. But when I changed to multiple GPUs, I encountered the following error. Btw, I was using PyTorch-lightening 1.8 instead of 1.4.

Traceback (most recent call last):
File "train_ddpm.py", line 710, in
trainer.fit(model, data)
File "/home/lix0i/miniconda3/envs/ldm_clone/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 603, in fit
call._call_and_handle_interrupt(
File "/home/lix0i/miniconda3/envs/ldm_clone/lib/python3.8/site-packages/pytorch_lightning/trainer/call.py", line 36, in _call_and_handle_interrupt
return trainer.strategy.launcher.launch(trainer_fn, *args, trainer=trainer, **kwargs)
File "/home/lix0i/miniconda3/envs/ldm_clone/lib/python3.8/site-packages/pytorch_lightning/strategies/launchers/multiprocessing.py", line 113, in launch
mp.start_processes(
File "/home/lix0i/miniconda3/envs/ldm_clone/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 179, in start_processes
process.start()
File "/home/lix0i/miniconda3/envs/ldm_clone/lib/python3.8/multiprocessing/process.py", line 121, in start
self._popen = self._Popen(self)
File "/home/lix0i/miniconda3/envs/ldm_clone/lib/python3.8/multiprocessing/context.py", line 284, in _Popen
return Popen(process_obj)
File "/home/lix0i/miniconda3/envs/ldm_clone/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 32, in init
super().init(process_obj)
File "/home/lix0i/miniconda3/envs/ldm_clone/lib/python3.8/multiprocessing/popen_fork.py", line 19, in init
self._launch(process_obj)
File "/home/lix0i/miniconda3/envs/ldm_clone/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 47, in _launch
reduction.dump(process_obj, fp)
File "/home/lix0i/miniconda3/envs/ldm_clone/lib/python3.8/multiprocessing/reduction.py", line 60, in dump
ForkingPickler(file, protocol).dump(obj)
AttributeError: Can't pickle local object 'always..inner'

[lx709]

problem solved

[ruairiseosamh]

How did you solve this? I am having the same issue.