TypeError: 'NoneType' object is not iterable`

[TechSheng]

Traceback (most recent call last):
File "train.py", line 60, in
main(args)
File "train.py", line 42, in main
trainer.fit(distiller)
File "/home/opu/anaconda3/envs/mobilestylegan/lib/python3.6/site-packages/pytorch_lightning/trainer/trainer.py", line 458, in fit
self._run(model)
File "/home/opu/anaconda3/envs/mobilestylegan/lib/python3.6/site-packages/pytorch_lightning/trainer/trainer.py", line 756, in _run
self.dispatch()
File "/home/opu/anaconda3/envs/mobilestylegan/lib/python3.6/site-packages/pytorch_lightning/trainer/trainer.py", line 797, in dispatch
self.accelerator.start_training(self)
File "/home/opu/anaconda3/envs/mobilestylegan/lib/python3.6/site-packages/pytorch_lightning/accelerators/accelerator.py", line 96, in start_training
self.training_type_plugin.start_training(trainer)
File "/home/opu/anaconda3/envs/mobilestylegan/lib/python3.6/site-packages/pytorch_lightning/plugins/training_type/training_type_plugin.py", line 144, in start_training
self._results = trainer.run_stage()
File "/home/opu/anaconda3/envs/mobilestylegan/lib/python3.6/site-packages/pytorch_lightning/trainer/trainer.py", line 807, in run_stage
return self.run_train()
File "/home/opu/anaconda3/envs/mobilestylegan/lib/python3.6/site-packages/pytorch_lightning/trainer/trainer.py", line 869, in run_train
self.train_loop.run_training_epoch()
File "/home/opu/anaconda3/envs/mobilestylegan/lib/python3.6/site-packages/pytorch_lightning/trainer/training_loop.py", line 489, in run_training_epoch
batch_output = self.run_training_batch(batch, batch_idx, dataloader_idx)
File "/home/opu/anaconda3/envs/mobilestylegan/lib/python3.6/site-packages/pytorch_lightning/trainer/training_loop.py", line 728, in run_training_batch
self.optimizer_step(optimizer, opt_idx, batch_idx, train_step_and_backward_closure)
File "/home/opu/anaconda3/envs/mobilestylegan/lib/python3.6/site-packages/pytorch_lightning/trainer/training_loop.py", line 432, in optimizer_step
using_lbfgs=is_lbfgs,
File "/home/opu/anaconda3/envs/mobilestylegan/lib/python3.6/site-packages/pytorch_lightning/core/lightning.py", line 1403, in optimizer_step
optimizer.step(closure=optimizer_closure)
File "/home/opu/anaconda3/envs/mobilestylegan/lib/python3.6/site-packages/pytorch_lightning/core/optimizer.py", line 214, in step
self.__optimizer_step(*args, closure=closure, profiler_name=profiler_name, **kwargs)
File "/home/opu/anaconda3/envs/mobilestylegan/lib/python3.6/site-packages/pytorch_lightning/core/optimizer.py", line 134, in __optimizer_step
trainer.accelerator.optimizer_step(optimizer, self._optimizer_idx, lambda_closure=closure, **kwargs)
File "/home/opu/anaconda3/envs/mobilestylegan/lib/python3.6/site-packages/pytorch_lightning/accelerators/accelerator.py", line 329, in optimizer_step
self.run_optimizer_step(optimizer, opt_idx, lambda_closure, **kwargs)
File "/home/opu/anaconda3/envs/mobilestylegan/lib/python3.6/site-packages/pytorch_lightning/accelerators/accelerator.py", line 336, in run_optimizer_step
self.training_type_plugin.optimizer_step(optimizer, lambda_closure=lambda_closure, **kwargs)
File "/home/opu/anaconda3/envs/mobilestylegan/lib/python3.6/site-packages/pytorch_lightning/plugins/training_type/training_type_plugin.py", line 193, in optimizer_step
optimizer.step(closure=lambda_closure, **kwargs)
File "/home/opu/anaconda3/envs/mobilestylegan/lib/python3.6/site-packages/torch/autograd/grad_mode.py", line 26, in decorate_context
return func(*args, **kwargs)
File "/home/opu/anaconda3/envs/mobilestylegan/lib/python3.6/site-packages/torch/optim/adam.py", line 66, in step
loss = closure()
File "/home/opu/anaconda3/envs/mobilestylegan/lib/python3.6/site-packages/pytorch_lightning/trainer/training_loop.py", line 723, in train_step_and_backward_closure
split_batch, batch_idx, opt_idx, optimizer, self.trainer.hiddens
File "/home/opu/anaconda3/envs/mobilestylegan/lib/python3.6/site-packages/pytorch_lightning/trainer/training_loop.py", line 813, in training_step_and_backward
result = self.training_step(split_batch, batch_idx, opt_idx, hiddens)
File "/home/opu/anaconda3/envs/mobilestylegan/lib/python3.6/site-packages/pytorch_lightning/trainer/training_loop.py", line 280, in training_step
training_step_output = self.trainer.accelerator.training_step(args)
File "/home/opu/anaconda3/envs/mobilestylegan/lib/python3.6/site-packages/pytorch_lightning/accelerators/accelerator.py", line 204, in training_step
return self.training_type_plugin.training_step(*args)
File "/home/opu/anaconda3/envs/mobilestylegan/lib/python3.6/site-packages/pytorch_lightning/plugins/training_type/training_type_plugin.py", line 155, in training_step
return self.lightning_module.training_step(*args, **kwargs)
File "/home/opu/MobileStyleGAN/core/distiller.py", line 70, in training_step
loss = self.generator_step(batch, batch_nb)
File "/home/opu/MobileStyleGAN/core/distiller.py", line 107, in generator_step
style, pred_t, gt_images = self.make_sample(batch)
File "/home/opu/anaconda3/envs/mobilestylegan/lib/python3.6/site-packages/torch/autograd/grad_mode.py", line 26, in decorate_context
return func(*args, **kwargs)
File "/home/opu/MobileStyleGAN/core/distiller.py", line 122, in make_sample
gt = self.synthesis_net(style)
File "/home/opu/anaconda3/envs/mobilestylegan/lib/python3.6/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/opu/MobileStyleGAN/core/models/synthesis_network.py", line 96, in forward
img = self.upsample(img) + rgb
File "/home/opu/anaconda3/envs/mobilestylegan/lib/python3.6/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/opu/MobileStyleGAN/core/models/modules/legacy.py", line 40, in forward
out = upfirdn2d(input, self.kernel, up=self.factor, down=1, pad=self.pad)
File "/home/opu/MobileStyleGAN/core/models/modules/ops/upfirdn2d.py", line 14, in upfirdn2d
input, kernel, (up, up), (down, down), (pad[0], pad[1], pad[0], pad[1])
File "/home/opu/MobileStyleGAN/core/models/modules/ops/upfirdn2d_cuda.py", line 99, in forward
ctx.save_for_backward(kernel, torch.flip(kernel, [0, 1]))
RuntimeError: CUDA error: an illegal memory access was encountered
Exception ignored in: <bound method tqdm.del of <pytorch_lightning.callbacks.progress.tqdm object at 0x7f944c6c4240>>
Traceback (most recent call last):
File "/home/opu/anaconda3/envs/mobilestylegan/lib/python3.6/site-packages/tqdm/std.py", line 1145, in del
File "/home/opu/anaconda3/envs/mobilestylegan/lib/python3.6/site-packages/tqdm/std.py", line 1299, in close
File "/home/opu/anaconda3/envs/mobilestylegan/lib/python3.6/site-packages/tqdm/std.py", line 1492, in display
File "/home/opu/anaconda3/envs/mobilestylegan/lib/python3.6/site-packages/tqdm/std.py", line 1148, in str
File "/home/opu/anaconda3/envs/mobilestylegan/lib/python3.6/site-packages/tqdm/std.py", line 1450, in format_dict
TypeError: 'NoneType' object is not iterable

when I tried to run the project, this error raised. I need help

[bes-dev]

@zuimeiyujianni hey, what batch size do you use?

RuntimeError: CUDA error: an illegal memory access was encountered

Usually it means that your batch size is too large

[TechSheng]

@bes-dev Thank you for your reply. I use the default setting batch_size = 2, and my device is a Tesla V100. I also tried batch_size =1, but the problem was raised again.

[bes-dev]

@zuimeiyujianni so, 'develop' works well on my side. I think that there are some errors with your environment (I haven't v100 GPUs and I can't test this setup :( ).

[TechSheng]

@bes-dev Thank you very much. I will continue checking my environment.