Insufficient shared memory or the value of the tensor for reference image goes to 0

[DeepanshuMakkar]

If I train the network with a default value of batch size and number or workers i get the error.

RuntimeError: invalid argument 0: Sizes of tensors must match except in dimension 0. Got 128 and 0 in dimension 2 at /pytorch/aten/src/TH/generic/THTensor.cpp:718

Even when I reduce the number of workers and batch size I receive this error, one of the column of tensor for reference image goes to 0.

Can you please help me with this error, i tried debugging but could not reach any solution I am using google colab's GPU.

Can you also please tell me what would be the minimum RAM requirement to train ESRGAN? Do you use multiple GPUS to train it? I am really excited to learn more about the network and train it but i am stuck at this problem.

[xinntao]

1. Can you give more error log? The problem is that you have different dimensions for the input tensors.
   The problem of one of the column of tensor for reference image goes to 0 is not from insufficient shared memory.
2. The tensor format is BCHW, so the dimension2 is H. It is strange why H is 0. You may go to the error line and print more information, like tensor size.
3. Did you mean GPU memory or RAM?
4. I used a single GPU to train, but it is really slow. Better with multi-GPU training.

[DeepanshuMakkar]

Yes I meant the GPU's RAM. These are the errors that i am getting with different values of no of workers, and batch size. Thanks a lot in advance.
Below is attached train log when I try to train with default values.
train (1).log

and this is the error that I receive:-

19-02-12 23:39:59.748 - INFO: Model [SRRaGANModel] is created.
19-02-12 23:39:59.749 - INFO: Start training...
ERROR: Unexpected bus error encountered in worker. This might be caused by insufficient shared memory (shm).
�Traceback (most recent call last):
File "/content/gdrive/My Drive/SISR/ESRGAN/BasicSR-master/codes/train.py", line 154, in
main()
File "/content/gdrive/My Drive/SISR/ESRGAN/BasicSR-master/codes/train.py", line 79, in main
for i, train_data in enumerate(train_loader):
File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py", line 631, in next
idx, batch = self._get_batch()
File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py", line 610, in _get_batch
return self.data_queue.get()
File "/usr/lib/python3.6/multiprocessing/queues.py", line 94, in get
res = self._recv_bytes()
File "/usr/lib/python3.6/multiprocessing/connection.py", line 216, in recv_bytes
buf = self._recv_bytes(maxlength)
File "/usr/lib/python3.6/multiprocessing/connection.py", line 407, in _recv_bytes
buf = self._recv(4)
File "/usr/lib/python3.6/multiprocessing/connection.py", line 379, in _recv
chunk = read(handle, remaining)
File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py", line 274, in handler
_error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 388) is killed by signal: Bus error.

If I reduce the number of workers from 8 to 1 then I get this error

19-02-12 23:48:49.544 - INFO: Model [SRRaGANModel] is created.
19-02-12 23:48:49.544 - INFO: Start training...
Traceback (most recent call last):
File "/content/gdrive/My Drive/SISR/ESRGAN/BasicSR-master/codes/train.py", line 154, in
main()
File "/content/gdrive/My Drive/SISR/ESRGAN/BasicSR-master/codes/train.py", line 79, in main
for i, train_data in enumerate(train_loader):
File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py", line 637, in next
return self._process_next_batch(batch)
File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py", line 658, in _process_next_batch
raise batch.exc_type(batch.exc_msg)
RuntimeError: Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py", line 138, in _worker_loop
samples = collate_fn([dataset[i] for i in batch_indices])
File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py", line 229, in default_collate
return {key: default_collate([d[key] for d in batch]) for key in batch[0]}
File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py", line 229, in
return {key: default_collate([d[key] for d in batch]) for key in batch[0]}
File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py", line 209, in default_collate
return torch.stack(batch, 0, out=out)
RuntimeError: invalid argument 0: Sizes of tensors must match except in dimension 0. Got 128 and 0 in dimension 2 at /pytorch/aten/src/TH/generic/THTensorMoreMath.cpp:1333

If i reduce the Batch size from 64 to 1 then i can notice the value of reference image going to zero after cetrain iterations as show below:-

19-02-12 23:59:44.272 - INFO: Model [SRRaGANModel] is created.
19-02-12 23:59:44.272 - INFO: Start training...
/usr/local/lib/python3.6/dist-packages/torch/nn/modules/upsampling.py:129: UserWarning: nn.Upsample is deprecated. Use nn.functional.interpolate instead.
warnings.warn("nn.{} is deprecated. Use nn.functional.interpolate instead.".format(self.name))
fake_H_list: [1, 3, 128, 128]
var_H_list: [1, 3, 128, 128]
v_list: [1, 3, 128, 128]
fake_H_list: [1, 3, 128, 128]
var_H_list: [1, 3, 128, 128]
v_list: [1, 3, 128, 128]
fake_H_list: [1, 3, 128, 128]
var_H_list: [1, 3, 128, 128]
v_list: [1, 3, 128, 128]
fake_H_list: [1, 3, 128, 128]
var_H_list: [1, 3, 128, 128]
v_list: [1, 3, 128, 128]
fake_H_list: [1, 3, 128, 128]
var_H_list: [1, 3, 128, 128]
v_list: [1, 3, 128, 128]
fake_H_list: [1, 3, 128, 128]
var_H_list: [1, 3, 128, 128]
v_list: [1, 3, 128, 128]
fake_H_list: [1, 3, 128, 128]
var_H_list: [1, 3, 128, 128]
v_list: [1, 3, 128, 128]
fake_H_list: [1, 3, 128, 128]
var_H_list: [1, 3, 128, 128]
v_list: [1, 3, 128, 128]
fake_H_list: [1, 3, 128, 128]
var_H_list: [1, 3, 128, 128]
v_list: [1, 3, 128, 128]
fake_H_list: [1, 3, 128, 128]
var_H_list: [1, 3, 128, 128]
v_list: [1, 3, 128, 128]
fake_H_list: [1, 3, 128, 128]
var_H_list: [1, 3, 128, 128]
v_list: [1, 3, 128, 128]
fake_H_list: [1, 3, 128, 128]
var_H_list: [1, 3, 128, 128]
v_list: [1, 3, 128, 128]
fake_H_list: [1, 3, 128, 128]
var_H_list: [1, 3, 128, 128]
v_list: [1, 3, 128, 128]
fake_H_list: [1, 3, 128, 128]
var_H_list: [1, 3, 128, 128]
v_list: [1, 3, 128, 128]
fake_H_list: [1, 3, 128, 128]
var_H_list: [1, 3, 128, 128]
v_list: [1, 3, 128, 128]
fake_H_list: [1, 3, 128, 128]
var_H_list: [1, 3, 128, 128]
v_list: [1, 3, 128, 128]
fake_H_list: [1, 3, 128, 128]
var_H_list: [1, 3, 128, 128]
v_list: [1, 3, 128, 128]
fake_H_list: [1, 3, 128, 128]
var_H_list: [1, 3, 128, 128]
v_list: [1, 3, 128, 128]
fake_H_list: [1, 3, 128, 128]
var_H_list: [1, 3, 128, 128]
v_list: [1, 3, 128, 128]
fake_H_list: [1, 3, 128, 128]
var_H_list: [1, 3, 128, 128]
v_list: [1, 3, 128, 128]
fake_H_list: [1, 3, 128, 128]
var_H_list: [1, 3, 128, 128]
v_list: [1, 3, 128, 128]
fake_H_list: [1, 3, 128, 128]
var_H_list: [1, 3, 128, 128]
v_list: [1, 3, 128, 128]
fake_H_list: [1, 3, 128, 128]
var_H_list: [1, 3, 128, 128]
v_list: [1, 3, 128, 128]
fake_H_list: [1, 3, 128, 128]
var_H_list: [1, 3, 128, 128]
v_list: [1, 3, 128, 128]
fake_H_list: [1, 3, 128, 128]
var_H_list: [1, 3, 128, 128]
v_list: [1, 3, 128, 128]
fake_H_list: [1, 3, 128, 128]
var_H_list: [1, 3, 128, 128]
v_list: [1, 3, 128, 128]
fake_H_list: [1, 3, 128, 128]
var_H_list: [1, 3, 128, 128]
v_list: [1, 3, 128, 128]
fake_H_list: [1, 3, 128, 128]
var_H_list: [1, 3, 128, 128]
v_list: [1, 3, 128, 128]
fake_H_list: [1, 3, 128, 128]
var_H_list: [1, 3, 128, 128]
v_list: [1, 3, 128, 128]
fake_H_list: [1, 3, 128, 128]
var_H_list: [1, 3, 128, 128]
v_list: [1, 3, 128, 128]
fake_H_list: [1, 3, 128, 128]
var_H_list: [1, 3, 128, 128]
v_list: [1, 3, 128, 128]
fake_H_list: [1, 3, 128, 128]
var_H_list: [1, 3, 128, 128]
v_list: [1, 3, 128, 128]
fake_H_list: [1, 3, 128, 128]
var_H_list: [1, 3, 0, 128]
v_list: [1, 3, 0, 128]
Traceback (most recent call last):
File "/content/gdrive/My Drive/SISR/ESRGAN/BasicSR-master/codes/train.py", line 154, in
main()
File "/content/gdrive/My Drive/SISR/ESRGAN/BasicSR-master/codes/train.py", line 88, in main
model.optimize_parameters(current_step)
File "/content/gdrive/My Drive/SISR/ESRGAN/BasicSR-master/codes/models/SRRaGAN_model.py", line 133, in optimize_parameters
l_g_pix = self.l_pix_w * self.cri_pix(self.fake_H, self.var_H)
File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 489, in call
result = self.forward(*input, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/loss.py", line 93, in forward
return F.l1_loss(input, target, reduction=self.reduction)
File "/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py", line 2134, in l1_loss
expanded_input, expanded_target = torch.broadcast_tensors(input, target)
File "/usr/local/lib/python3.6/dist-packages/torch/functional.py", line 50, in broadcast_tensors
return torch._C._VariableFunctions.broadcast_tensors(tensors)
RuntimeError: The size of tensor a (128) must match the size of tensor b (0) at non-singleton dimension 2

[xinntao]

@DeepanshuMakkar
For the first error: ERROR: Unexpected bus error encountered in worker. This might be caused by insufficient shared memory (shm). I have not encountered this error. Try to google it~

The 2nd and 3rd problems Got 128 and 0 in dimension 2 and The size of tensor a (128) must match the size of tensor b (0) at non-singleton dimension 2, look like the same problem.
Have you checked your data? Whether if each image has the proper size?

You can also print the image name. See which image cause this problem and check that image.

[DeepanshuMakkar]

Hello @xinntao you were right, the data was getting corrupted when I was doing the preprocessing on google colab for some reason. Also i had to increase the shared memory and it works now. But i have one issue, i am traning the nework on images with radial distrotion and the network is not raning that well, it has already been 120000 iterations and still the PSNR has just gone to 9.2, and it fluctuating between 8 and 9. Does this mean that the network is stuck at a local minimum? what hyper parameter would you suggest to imrove the results? Thanks in advance.

[xinntao]

@DeepanshuMakkar
There must be something wrong if the PSNR is just 9.2. Have you checked your outputs?
What if finetuning from a pre-trained network? (even trained from other datasets, like DIV2K) ?
(I used the hyperparameters in the configuration.json files. Maybe it is not optimal for your task. But I think it at least will produce some not bad results)

[taily-khucnaykhongquantrong]

@DeepanshuMakkar How could you increase the shm? Google colab do not let me do that. One weird thing that the smh is 18446744073692774399kB ~ 18446744.1 petabytes, how could that is insufficient? :(