Issues with Loss1 + 0*Loss2 and graph computation

[WendyShang]

Hello,

I found some absurd errors when trying to replicate my luatorch code to pytorch. After some serious debugging, my intuition tells me it goes down to some intrinsic parts of PyTorch.

I run on master brunch; to run the code I provided, place it into examples/vae/ folder, then run python xxx.py to reproduce the errors.

I attached a few modifications of examples/vae/main.py in the zip file pytorch_debug.zip

(1) main_conv_new_rep.py: this code runs, but note that I modified the reparametrization part and got rid of KLD error.

(2) main_conv_new_rep_KLD.py: this code DOES NOT run, and gives the error message (below) with the only difference between the previous code to be KLD (line 144),I set KLD to be 0. set KLD to be 0.**

oat, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type>, Dtype = float, Acctype = float]: block: [65,0,0], thread: [59,0,0] Assertion `input >= 0. && input <= 1.` failed.
/home/shangw/pytorch/torch/lib/THCUNN/BCECriterion.cu:30: Acctype bce_functor<Dtype, Acctype>::operator()(Tuple) [with Tuple = thrust::tuple<float, float, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type>, Dtype = float, Acctype = float]: block: [65,0,0], thread: [60,0,0] Assertion `input >= 0. && input <= 1.` failed.
/home/shangw/pytorch/torch/lib/THCUNN/BCECriterion.cu:30: Acctype bce_functor<Dtype, Acctype>::operator()(Tuple) [with Tuple = thrust::tuple<float, float, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type>, Dtype = float, Acctype = float]: block: [65,0,0], thread: [61,0,0] Assertion `input >= 0. && input <= 1.` failed.
/home/shangw/pytorch/torch/lib/THCUNN/BCECriterion.cu:30: Acctype bce_functor<Dtype, Acctype>::operator()(Tuple) [with Tuple = thrust::tuple<float, float, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type>, Dtype = float, Acctype = float]: block: [65,0,0], thread: [62,0,0] Assertion `input >= 0. && input <= 1.` failed.
/home/shangw/pytorch/torch/lib/THCUNN/BCECriterion.cu:30: Acctype bce_functor<Dtype, Acctype>::operator()(Tuple) [with Tuple = thrust::tuple<float, float, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type>, Dtype = float, Acctype = float]: block: [65,0,0], thread: [63,0,0] Assertion `input >= 0. && input <= 1.` failed.
CUDA error after cudaEventDestroy in future dtor: device-side assert triggeredTraceback (most recent call last):
  File "main_conv_new_rep_kl.py", line 188, in <module>
    train(epoch)
  File "main_conv_new_rep_kl.py", line 159, in train
    loss = loss_function(recon_batch, data, mu, logvar)
  File "main_conv_new_rep_kl.py", line 135, in loss_function
    BCE = reconstruction_function(recon_x, x)
  File "/home/shangw/local/anaconda3/envs/py35s/lib/python3.5/site-packages/torch/nn/modules/module.py", line 225, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/shangw/local/anaconda3/envs/py35s/lib/python3.5/site-packages/torch/nn/modules/loss.py", line 34, in forward
    return backend_fn(self.size_average, weight=self.weight)(input, target)
  File "/home/shangw/local/anaconda3/envs/py35s/lib/python3.5/site-packages/torch/nn/_functions/thnn/loss.py", line 28, in forward
    result = super(BCELoss, self).forward(input, target)
  File "/home/shangw/local/anaconda3/envs/py35s/lib/python3.5/site-packages/torch/nn/_functions/thnn/auto.py", line 41, in forward
    output, *self.additional_args)
RuntimeError: cudaEventSynchronize in future::wait: device-side assert triggered

(3) main_conv_old_rep.py: this code DOES NOT run, gives the same error, and the only difference between this and the first one main_conv_new_rep.py is that I used the original reparametrization function (line 120), which is mathematically equivalent (and should be computationally equivalent as well).

If someone can possibly look into this, I would greatly appreciate the effort! since it is a little bit time sensitive... Thanks!

[hongyi-zhang]

Maybe you could try checking out an earlier commit, or the stable release 0.1.12_2? I can see the master brunch is fast-evolving towards v0.2 to add some new features, but maybe you don't need them to transplant your code.

[apaszke]

BCELoss accepts only inputs that have all elements in range [0; 1] but this condition doesn't hold in your case

[WendyShang]

I did have sigmoid at the end. It didn't hold because the inputs (or outputs from the network) have nan. I have attached a version that does work by commenting out the KL loss, despite the error comes from BCE loss.

[soumith]

@WendyShang adding the KL Loss might be making some of the weights nan (which might be making the next iteration of BCELoss get nan targets). Now, the reason KL loss might be generating NaNs, I'm not entirely sure, but 0 * KLLoss will still propagate nans back in the gradients. (0 * nan = nan)

[WendyShang]

I agree except that if I run with only KL loss and comment out BCE loss it also works.
So basically:

KL  #ok
BCE #ok
0*KL + BCE #nope
KL + 0*BCE #nope

In addition there is that reparametrization part (i.e. before I re-write that part of the code into an equivalent format, it didn't run).

[WendyShang]

I finally figured out a walkaround after trying many ways to debug. So the way I found to solve this problem is to adjust the batchnorm layer hyperparameters. Originally, I have epsilon = 1e-6 and momentum = 0.9; now I switched to the default value and it allows training (still waiting for final result, but looks promising so far). Another way is to perform gradient clipping but the quantitative results are very suboptimal. Thanks for all your help and suggestions! especially @justin.

[anuragranj]

Hi @WendyShang, I am having exactly the same problem. I am running the fully connected vae network (non-convolutional) without any batchnorm. Do you have any suggestions?

[anuragranj]

So I replaced the non-linearity in the encoder from relu to sigmoid, and it works.
However, that should not happen, since the output non-linearity has always been a sigmoid. I am not sure how the loss MSELoss(output, label) between output and labels is being influenced by latent variable, z in the network and causing this error.

[WendyShang]

Hi, @anuragranj , my conjecture is numerical issue, that is, the handling of numerical instability. for example, in your case, ReLU can much more easily blow up on the positive end but sigmoid is always constrained between 0 and 1. Similarly, in my case, I changed the eps for BN from 1e-6 to 1e-5, where 1e-6 works with Torch but PyTorch seems to need a much bigger buffer to prevent numerical instability.

If you share your code privately with me, I can take a look at any possible walkaround :) but i am not sure if there is a universal fix for it.

[linrio]

I have met the same problem by setting the loss function with nn.BCELoss(). It throws following error:

/opt/conda/conda-bld/pytorch_1525909934016/work/aten/src/THCUNN/BCECriterion.cu:30: Acctype bce_functor<Dtype, Acctype>::operator()(Tuple) [with Tuple = thrust::detail::tuple_of_iterator_references<thrust::device_reference<float>, thrust::device_reference<float>, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type>, Dtype = float, Acctype = float]: block: [1,0,0], thread: [128,0,0] Assertion `input >= 0. && input <= 1.` failed.
/opt/conda/conda-bld/pytorch_1525909934016/work/aten/src/THCUNN/BCECriterion.cu:30: Acctype bce_functor<Dtype, Acctype>::operator()(Tuple) [with Tuple = thrust::detail::tuple_of_iterator_references<thrust::device_reference<float>, thrust::device_reference<float>, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type>, Dtype = float, Acctype = float]: block: [1,0,0], thread: [73,0,0] Assertion `input >= 0. && input <= 1.` failed.
/opt/conda/conda-bld/pytorch_1525909934016/work/aten/src/THCUNN/BCECriterion.cu:30: Acctype bce_functor<Dtype, Acctype>::operator()(Tuple) [with Tuple = thrust::detail::tuple_of_iterator_references<thrust::device_reference<float>, thrust::device_reference<float>, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type>, Dtype = float, Acctype = float]: block: [3,0,0], thread: [14,0,0] Assertion `input >= 0. && input <= 1.` failed.
/opt/conda/conda-bld/pytorch_1525909934016/work/aten/src/THCUNN/BCECriterion.cu:30: Acctype bce_functor<Dtype, Acctype>::operator()(Tuple) [with Tuple = thrust::detail::tuple_of_iterator_references<thrust::device_reference<float>, thrust::device_reference<float>, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type>, Dtype = float, Acctype = float]: block: [1,0,0], thread: [221,0,0] Assertion `input >= 0. && input <= 1.` failed.
/opt/conda/conda-bld/pytorch_1525909934016/work/aten/src/THCUNN/BCECriterion.cu:30: Acctype bce_functor<Dtype, Acctype>::operator()(Tuple) [with Tuple = thrust::detail::tuple_of_iterator_references<thrust::device_reference<float>, thrust::device_reference<float>, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type>, Dtype = float, Acctype = float]: block: [0,0,0], thread: [4,0,0] Assertion `input >= 0. && input <= 1.` failed.
/opt/conda/conda-bld/pytorch_1525909934016/work/aten/src/THCUNN/BCECriterion.cu:30: Acctype bce_functor<Dtype, Acctype>::operator()(Tuple) [with Tuple = thrust::detail::tuple_of_iterator_references<thrust::device_reference<float>, thrust::device_reference<float>, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type>, Dtype = float, Acctype = float]: block: [2,0,0], thread: [11,0,0] Assertion `input >= 0. && input <= 1.` failed.
/opt/conda/conda-bld/pytorch_1525909934016/work/aten/src/THCUNN/BCECriterion.cu:30: Acctype bce_functor<Dtype, Acctype>::operator()(Tuple) [with Tuple = thrust::detail::tuple_of_iterator_references<thrust::device_reference<float>, thrust::device_reference<float>, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type>, Dtype = float, Acctype = float]: block: [2,0,0], thread: [70,0,0] Assertion `input >= 0. && input <= 1.` failed.
/opt/conda/conda-bld/pytorch_1525909934016/work/aten/src/THCUNN/BCECriterion.cu:30: Acctype bce_functor<Dtype, Acctype>::operator()(Tuple) [with Tuple = thrust::detail::tuple_of_iterator_references<thrust::device_reference<float>, thrust::device_reference<float>, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type>, Dtype = float, Acctype = float]: block: [2,0,0], thread: [202,0,0] Assertion `input >= 0. && input <= 1.` failed.
/opt/conda/conda-bld/pytorch_1525909934016/work/aten/src/THCUNN/BCECriterion.cu:30: Acctype bce_functor<Dtype, Acctype>::operator()(Tuple) [with Tuple = thrust::detail::tuple_of_iterator_references<thrust::device_reference<float>, thrust::device_reference<float>, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type>, Dtype = float, Acctype = float]: block: [2,0,0], thread: [179,0,0] Assertion `input >= 0. && input <= 1.` failed.
/opt/conda/conda-bld/pytorch_1525909934016/work/aten/src/THCUNN/BCECriterion.cu:30: Acctype bce_functor<Dtype, Acctype>::operator()(Tuple) [with Tuple = thrust::detail::tuple_of_iterator_references<thrust::device_reference<float>, thrust::device_reference<float>, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type>, Dtype = float, Acctype = float]: block: [2,0,0], thread: [101,0,0] Assertion `input >= 0. && input <= 1.` failed.
/opt/conda/conda-bld/pytorch_1525909934016/work/aten/src/THCUNN/BCECriterion.cu:30: Acctype bce_functor<Dtype, Acctype>::operator()(Tuple) [with Tuple = thrust::detail::tuple_of_iterator_references<thrust::device_reference<float>, thrust::device_reference<float>, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type>, Dtype = float, Acctype = float]: block: [2,0,0], thread: [117,0,0] Assertion `input >= 0. && input <= 1.` failed.
/opt/conda/conda-bld/pytorch_1525909934016/work/aten/src/THCUNN/BCECriterion.cu:30: Acctype bce_functor<Dtype, Acctype>::operator()(Tuple) [with Tuple = thrust::detail::tuple_of_iterator_references<thrust::device_reference<float>, thrust::device_reference<float>, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type>, Dtype = float, Acctype = float]: block: [2,0,0], thread: [134,0,0] Assertion `input >= 0. && input <= 1.` failed.
/opt/conda/conda-bld/pytorch_1525909934016/work/aten/src/THCUNN/BCECriterion.cu:30: Acctype bce_functor<Dtype, Acctype>::operator()(Tuple) [with Tuple = thrust::detail::tuple_of_iterator_references<thrust::device_reference<float>, thrust::device_reference<float>, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type>, Dtype = float, Acctype = float]: block: [0,0,0], thread: [167,0,0] Assertion `input >= 0. && input <= 1.` failed.
/opt/conda/conda-bld/pytorch_1525909934016/work/aten/src/THCUNN/BCECriterion.cu:30: Acctype bce_functor<Dtype, Acctype>::operator()(Tuple) [with Tuple = thrust::detail::tuple_of_iterator_references<thrust::device_reference<float>, thrust::device_reference<float>, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type>, Dtype = float, Acctype = float]: block: [0,0,0], thread: [137,0,0] Assertion `input >= 0. && input <= 1.` failed.
THCudaCheck FAIL file=/opt/conda/conda-bld/pytorch_1525909934016/work/aten/src/THC/THCCachingHostAllocator.cpp line=257 error=59 : device-side assert triggered
Traceback (most recent call last):
  File "./ncf.py", line 234, in <module>
    main()
  File "./ncf.py", line 204, in main
    loss = criterion(outputs, label)
  File "/home/linlf/anaconda3/envs/venv3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 491, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/linlf/anaconda3/envs/venv3/lib/python3.6/site-packages/torch/nn/modules/loss.py", line 433, in forward
    reduce=self.reduce)
  File "/home/linlf/anaconda3/envs/venv3/lib/python3.6/site-packages/torch/nn/functional.py", line 1483, in binary_cross_entropy
    return torch._C._nn.binary_cross_entropy(input, target, weight, size_average, reduce)
RuntimeError: reduce failed to synchronize: device-side assert triggered
Finish training in 406 seconds
ENDING TIMING RUN AT 2018-05-27 01:04:36 PM
RESULT,recommendation,1,1122,linlf,2018-05-27 12:45:54 PM

And I replace the nn.BCELoss() loss funtion by nn.BCEWithLogitsLoss().
Done!