~/GSCNN$ python train.py --evaluate --snapshot checkpoints/best_cityscapes_checkpoint.pth
08-29 01:56:42.146 train fine cities: ['train/aachen', 'train/bochum', 'train/bremen', 'train/cologne', 'train/darmstadt', 'train/dusseldorf', 'train/erfurt', 'train/hamburg', 'train/hanover', 'train/jena', 'train/krefeld', 'train/monchengladbach', 'train/strasbourg', 'train/stuttgart', 'train/tubingen', 'train/ulm', 'train/weimar', 'train/zurich']
08-29 01:56:42.154 Cityscapes-train: 2975 images
08-29 01:56:42.154 val fine cities: ['val/munster', 'val/frankfurt', 'val/lindau']
08-29 01:56:42.155 Cityscapes-val: 500 images
08-29 01:56:42.156 Using Per Image based weighted loss
/home/tets/p36pytorch1/lib/python3.6/site-packages/torch/nn/modules/loss.py:216: UserWarning: NLLLoss2d has been deprecated. Please use NLLLoss instead as a drop-in replacement and see https://pytorch.org/docs/master/nn.html#torch.nn.NLLLoss for more details.
  warnings.warn("NLLLoss2d has been deprecated. "
/home/tets/p36pytorch1/lib/python3.6/site-packages/torch/nn/_reduction.py:46: UserWarning: size_average and reduce args will be deprecated, please use reduction='mean' instead.
  warnings.warn(warning.format(ret))
08-29 01:56:42.156 Using Cross Entropy Loss
/home/tets/p36pytorch1/lib/python3.6/site-packages/encoding/nn/syncbn.py:149: EncodingDeprecationWarning: encoding.nn.BatchNorm2d is now deprecated in favor of encoding.nn.SyncBatchNorm.
  .format('BatchNorm2d', SyncBatchNorm.__name__), EncodingDeprecationWarning)
08-29 01:56:44.793 Model params = 137.3M
08-29 01:56:44.813 Loading weights from model checkpoints/best_cityscapes_checkpoint.pth
08-29 01:56:45.101 Load Compelete
Traceback (most recent call last):
  File "train.py", line 382, in <module>
    main()
  File "train.py", line 142, in main
    optim, 1, writer) #replacig epoch variable by 1 given we are just evaluing and that epoch is not defined (bug)
  File "train.py", line 301, in validate
    seg_out, edge_out = net(input)    # output = (1, 19, 713, 713)
  File "/home/tets/p36pytorch1/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/tets/p36pytorch1/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 150, in forward
    return self.module(*inputs[0], **kwargs[0])
  File "/home/tets/p36pytorch1/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/tets/GSCNN/network/gscnn.py", line 272, in forward
    m7 = self.mod7(m6) 
  File "/home/tets/p36pytorch1/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/tets/p36pytorch1/lib/python3.6/site-packages/torch/nn/modules/container.py", line 92, in forward
    input = module(input)
  File "/home/tets/p36pytorch1/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/tets/GSCNN/network/wider_resnet.py", line 175, in forward
    out = self.convs(bn1)
  File "/home/tets/p36pytorch1/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/tets/p36pytorch1/lib/python3.6/site-packages/torch/nn/modules/container.py", line 92, in forward
    input = module(input)
  File "/home/tets/p36pytorch1/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/tets/p36pytorch1/lib/python3.6/site-packages/torch/nn/modules/container.py", line 92, in forward
    input = module(input)
  File "/home/tets/p36pytorch1/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/tets/p36pytorch1/lib/python3.6/site-packages/encoding/nn/syncbn.py", line 122, in forward
    self.activation, self.slope).view(input_shape)
  File "/home/tets/p36pytorch1/lib/python3.6/site-packages/encoding/functions/syncbn.py", line 95, in forward
    y = lib.gpu.batchnorm_forward(x, _ex, _exs, gamma, beta, ctx.eps)
RuntimeError: CUDA out of memory. Tried to allocate 256.00 MiB (GPU 0; 5.94 GiB total capacity; 4.98 GiB already allocated; 80.75 MiB free; 141.14 MiB cached) (malloc at /pytorch/c10/cuda/CUDACachingAllocator.cpp:267)
frame #0: std::function<std::string ()>::operator()() const + 0x11 (0x7fc296466441 in /home/tets/p36pytorch1/lib/python3.6/site-packages/torch/lib/libc10.so)
frame #1: c10::Error::Error(c10::SourceLocation, std::string const&) + 0x2a (0x7fc296465d7a in /home/tets/p36pytorch1/lib/python3.6/site-packages/torch/lib/libc10.so)
frame #2: <unknown function> + 0x15bc0 (0x7fc293f55bc0 in /home/tets/p36pytorch1/lib/python3.6/site-packages/torch/lib/libc10_cuda.so)
frame #3: <unknown function> + 0x16247 (0x7fc293f56247 in /home/tets/p36pytorch1/lib/python3.6/site-packages/torch/lib/libc10_cuda.so)
frame #4: at::native::empty_cuda(c10::ArrayRef<long>, c10::TensorOptions const&) + 0x121 (0x7fc2a3534f81 in /home/tets/p36pytorch1/lib/python3.6/site-packages/torch/lib/libcaffe2_gpu.so)
frame #5: at::CUDAType::empty(c10::ArrayRef<long>, c10::TensorOptions const&) const + 0x19b (0x7fc2a217c6fb in /home/tets/p36pytorch1/lib/python3.6/site-packages/torch/lib/libcaffe2_gpu.so)
frame #6: torch::autograd::VariableType::empty(c10::ArrayRef<long>, c10::TensorOptions const&) const + 0x284 (0x7fc294e1f094 in /home/tets/p36pytorch1/lib/python3.6/site-packages/torch/lib/libtorch.so.1)
frame #7: at::native::zeros(c10::ArrayRef<long>, c10::TensorOptions const&) + 0x40 (0x7fc297092a40 in /home/tets/p36pytorch1/lib/python3.6/site-packages/torch/lib/libcaffe2.so)
frame #8: at::native::zeros_like(at::Tensor const&, c10::TensorOptions const&) + 0x3b (0x7fc297092adb in /home/tets/p36pytorch1/lib/python3.6/site-packages/torch/lib/libcaffe2.so)
frame #9: at::native::zeros_like(at::Tensor const&) + 0x2b (0x7fc297092bfb in /home/tets/p36pytorch1/lib/python3.6/site-packages/torch/lib/libcaffe2.so)
frame #10: at::TypeDefault::zeros_like(at::Tensor const&) const + 0x55 (0x7fc297360945 in /home/tets/p36pytorch1/lib/python3.6/site-packages/torch/lib/libcaffe2.so)
frame #11: torch::autograd::VariableType::zeros_like(at::Tensor const&) const + 0x241 (0x7fc294f19bf1 in /home/tets/p36pytorch1/lib/python3.6/site-packages/torch/lib/libtorch.so.1)
frame #12: <unknown function> + 0xbcf18 (0x7fc27eb31f18 in /home/tets/p36pytorch1/lib/python3.6/site-packages/encoding/lib/gpu/enclib_gpu.so)
frame #13: BatchNorm_Forward_CUDA(at::Tensor, at::Tensor, at::Tensor, at::Tensor, at::Tensor, float) + 0x68 (0x7fc27eb326cf in /home/tets/p36pytorch1/lib/python3.6/site-packages/encoding/lib/gpu/enclib_gpu.so)
frame #14: <unknown function> + 0x9affc (0x7fc27eb0fffc in /home/tets/p36pytorch1/lib/python3.6/site-packages/encoding/lib/gpu/enclib_gpu.so)
frame #15: <unknown function> + 0x946a2 (0x7fc27eb096a2 in /home/tets/p36pytorch1/lib/python3.6/site-packages/encoding/lib/gpu/enclib_gpu.so)
frame #16: <unknown function> + 0x8c4c4 (0x7fc27eb014c4 in /home/tets/p36pytorch1/lib/python3.6/site-packages/encoding/lib/gpu/enclib_gpu.so)
frame #17: <unknown function> + 0x8c7e1 (0x7fc27eb017e1 in /home/tets/p36pytorch1/lib/python3.6/site-packages/encoding/lib/gpu/enclib_gpu.so)
frame #18: <unknown function> + 0x79c41 (0x7fc27eaeec41 in /home/tets/p36pytorch1/lib/python3.6/site-packages/encoding/lib/gpu/enclib_gpu.so)
frame #19: python() [0x4f8925]
<omitting python frames>
frame #21: python() [0x4f6128]
frame #23: python() [0x5a1481]
frame #25: THPFunction_apply(_object*, _object*) + 0x6b1 (0x7fc2d5c00481 in /home/tets/p36pytorch1/lib/python3.6/site-packages/torch/lib/libtorch_python.so)
frame #26: python() [0x4f858d]
frame #29: python() [0x5a1481]
frame #32: python() [0x4f6128]
frame #34: python() [0x5a1481]
frame #36: python() [0x513601]
frame #38: python() [0x4f88ba]
frame #41: python() [0x5a1481]
frame #44: python() [0x4f6128]
frame #46: python() [0x5a1481]
frame #48: python() [0x513601]
frame #50: python() [0x4f88ba]
frame #53: python() [0x5a1481]
frame #56: python() [0x4f6128]
frame #58: python() [0x5a1481]
frame #60: python() [0x513601]
frame #62: python() [0x4f88ba]