narrow tensor to zero elements fails

[yhlleo]

When ran the code on a single GPU, I got an error:

...
RuntimeError: start (0) + length (0) exceeds dimension size (0). (narrow at /opt/conda/conda-bld/pytorch_1533672544752/work/aten/src/ATen/native/TensorShape.cpp:157)
frame #0: at::Type::narrow(at::Tensor const&, long, long, long) const + 0x80 (0x7f5e4df3de80 in /data/yahui/anaconda2/envs/pytorch-CycleGAN-and-pix2pix/lib/python3.5/site-packages/torch/lib/libcaffe2.so)
...

I have checked the installation and configuration, both of them are correct.
So, do you have any ideas for the problem?

[junyanz]

Could you give more details? Which dataset did you use? What is the command line?

[yhlleo]

Thanks for your responses.

I ran the code on edge2shoes dataset using the command line:

sh ./scripts/train_edges2shoes.sh

I didn't change any configuration of training. Here are more details on the error:

...
---------- Networks initialized -------------
[Network G] Total number of parameters : 54.793 M
[Network D] Total number of parameters : 3.459 M
[Network D2] Total number of parameters : 3.459 M
[Network E] Total number of parameters : 2.590 M
-----------------------------------------------
saving the latest model (epoch 1, total_iters 10000)
saving the latest model (epoch 1, total_iters 20000)
saving the latest model (epoch 1, total_iters 30000)
saving the latest model (epoch 1, total_iters 40000)
Traceback (most recent call last):
  File "./train.py", line 45, in <module>
    model.optimize_parameters()   # calculate loss functions, get gradients, update network weights
  File "/data/xxx/CODES/BicycleGAN/models/bicycle_gan_model.py", line 206, in optimize_parameters
    self.update_D()
  File "/data/xxx/CODES/BicycleGAN/models/bicycle_gan_model.py", line 177, in update_D
...
File "/data/xxx/anaconda2/envs/pytorch-CycleGAN-and-pix2pix/lib/python3.5/site-packages/torch/cuda/comm.py", line 142, in scatter
    return tuple(torch._C._scatter(tensor, devices, chunk_sizes, dim, streams))
RuntimeError: start (0) + length (0) exceeds dimension size (0). (narrow at /opt/conda/conda-bld/pytorch_1533672544752/work/aten/src/ATen/native/TensorShape.cpp:157)
frame #0: at::Type::narrow(at::Tensor const&, long, long, long) const + 0x80 (0x7fd18ce50e80 in /data/xxx/anaconda2/envs/pytorch-CycleGAN-and-pix2pix/lib/python3.5/site-packages/torch/lib/libcaffe2.so)
frame #1: at::native::split_with_sizes(at::Tensor const&, at::ArrayRef<long>, long) + 0x18e (0x7fd18ccc20be in /data/xxx/anaconda2/envs/pytorch-CycleGAN-and-pix2pix/lib/python3.5/site-packages/torch/lib/libcaffe2.so)
frame #2: at::Type::split_with_sizes(at::Tensor const&, at::ArrayRef<long>, long) const + 0x80 (0x7fd18ce54620 in /data/xxx/anaconda2/envs/pytorch-CycleGAN-and-pix2pix/lib/python3.5/site-packages/torch/lib/libcaffe2.so)
frame #3: torch::autograd::VariableType::split_with_sizes(at::Tensor const&, at::ArrayRef<long>, long) const + 0x22b (0x7fd18ecd990b in /data/xxx/anaconda2/envs/pytorch-CycleGAN-and-pix2pix/lib/python3.5/site-packages/torch/_C.cpython-35m-x86_64-linux-gnu.so)
frame #4: at::native::chunk(at::Tensor const&, long, long) + 0x1a3 (0x7fd18ccc6083 in /data/xxx/anaconda2/envs/pytorch-CycleGAN-and-pix2pix/lib/python3.5/site-packages/torch/lib/libcaffe2.so)
frame #5: at::Type::chunk(at::Tensor const&, long, long) const + 0x78 (0x7fd18ce53e18 in /data/xxx/anaconda2/envs/pytorch-CycleGAN-and-pix2pix/lib/python3.5/site-packages/torch/lib/libcaffe2.so)
frame #6: torch::autograd::VariableType::chunk(at::Tensor const&, long, long) const + 0x1b1 (0x7fd18ecb60b1 in /data/xxx/anaconda2/envs/pytorch-CycleGAN-and-pix2pix/lib/python3.5/site-packages/torch/_C.cpython-35m-x86_64-linux-gnu.so)
frame #7: torch::cuda::scatter(at::Tensor const&, at::ArrayRef<long>, at::optional<std::vector<long, std::allocator<long> > > const&, long, at::optional<std::vector<CUDAStreamInternals*, std::allocator<CUDAStreamInternals*> > > const&) + 0x1151 (0x7fd18efe4ad1 in /data/xxx/anaconda2/envs/pytorch-CycleGAN-and-pix2pix/lib/python3.5/site-packages/torch/_C.cpython-35m-x86_64-linux-gnu.so)
frame #8: <unknown function> + 0xba586b (0x7fd18efea86b in /data/xxx/anaconda2/envs/pytorch-CycleGAN-and-pix2pix/lib/python3.5/site-packages/torch/_C.cpython-35m-x86_64-linux-gnu.so)
frame #9: <unknown function> + 0x353609 (0x7fd18e798609 in /data/xxx/anaconda2/envs/pytorch-CycleGAN-and-pix2pix/lib/python3.5/site-packages/torch/_C.cpython-35m-x86_64-linux-gnu.so)
<omitting python frames>
frame #17: THPFunction_apply(_object*, _object*) + 0x412 (0x7fd18eb256e2 in /data/xxx/anaconda2/envs/pytorch-CycleGAN-and-pix2pix/lib/python3.5/site-packages/torch/_C.cpython-35m-x86_64-linux-gnu.so)
frame #53: __libc_start_main + 0xf5 (0x7fd1a0d5cec5 in /lib/x86_64-linux-gnu/libc.so.6)

And, my GPU is one Tesla K40 with 12GB memory.

[junyanz]

I recently updated the code. Could you try the latest code ?

[yhlleo]

OK, thanks. Once the error is gone, I'll let you know.

[yhlleo]

I have tried to run the project on another machine with a TITAN X GPU. It was exciting that the codes ran well. So, I think there may be some problems of the hardware. I'm so sorry to bother you.
Best regards.

[junyanz]

No problem I am glad that it works for you.