sparse_softmax_cross_entropy_with_logits returns NaN instead of raising an error when the class int is not in the range [0, logit_size) when run on GPU

[w4nderlust]

If you provide to tf.nn.tf.nn.sparse_softmax_cross_entropy_with_logits() labels that are not inside the number of classes (that is the length of the logits, or the length of the second dimension of the logits if using the first dimension for batching), you get a loss of NaN and then everything in the model becomes NaN. I spent quite some time debugging this in my model and I believe that this should not be silent, there should be at least a warning at execution time.

More detail I just discovered: if you run it on CPU an InvalidAgrumentError is raised, if you run it on GPU, you get NaN, so you may want to fix only the GPU implementation to behave like the CPU one.

Environment info

Operating System: Ubuntu 16.04 64bit
Installed version of CUDA and cuDNN: cuda 8.0 cudnn 5.1
Tensorflow version: 1.0.0

If possible, provide a minimal reproducible example (We usually don't have time to read hundreds of lines of your code)

Run it with CUDA_VISIBLE_DEVICE="" to see the difference on CPU and GPU.

import numpy as np
import tensorflow as tf

classes = 5
num_datapoints = 100

xs = np.random.rand(num_datapoints,4)
ys = np.random.randint(classes + 1, size=num_datapoints)

graph = tf.Graph()
with graph.as_default():
    x = tf.placeholder(tf.float32, shape=[None, 4])
    y = tf.placeholder(tf.int32, shape=[None, ])

    w = tf.Variable(tf.random_normal([4, classes]))
    b = tf.Variable(tf.random_normal([classes]))

    logits = tf.matmul(x, w) + b

    loss = tf.reduce_mean(tf.nn.sparse_softmax_cross_entropy_with_logits(logits=logits, labels=y))
    optimizer = tf.train.AdamOptimizer(0.01).minimize(loss)


with tf.Session(graph=graph) as session:
    tf.global_variables_initializer().run()
    for step in range(201):
        _, loss_val = session.run(
            [optimizer, loss],
            feed_dict={x: xs, y: ys})
        print("step {step} - loss: {loss_val:.6f}".format(step=step, loss_val=loss_val))

[aselle]

@zheng-xq, is this intended behavior? Is there any way to make this more friendly?

[zheng-xq]

@ebrevdo, who adds the fused implementation of that kernel

[ebrevdo]

I'm afraid it's not possible to efficiently send back error messages on invalid inputs with GPU. Other ops behave similarly, returning Nan when an error condition occurs on GPU.

…

On Mar 24, 2017 10:42 AM, "zheng-xq" ***@***.***> wrote: @ebrevdo <https://github.com/ebrevdo>, who adds the fused implementation of that kernel — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#8484 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABtim9OlP2f4EBMvVRvWgpGPD06nMesgks5rpAB2gaJpZM4MgDu4> .

[w4nderlust]

embedding_lookup has a similar behavior: returns 0 for indices outside the range on GPU while it complains about those indices on CPU. What I personally would like to see as a user is an homogeneous behavior. Something like returning 0 or null for all problems for which the CPU implementation returns an error or a warning and the GPU implementation can't do checks.
An additional idea / suggestion: maybe a "debug mode" flag could help, of by default, that you can turn on while developing and gives you more warning. Or, in alternative, write somewhere "we suggest to run everything on CPU while in development in order to get significant errors and warnings and to run models on GPU only after having debugged them on the CPU". I guess transparency and explanations about why something like this happens is the best approach.

[girving]

Closing for now since this needs the same unimplemented mechanism for GPU error reporting that #2594 and #3638 require.

[kartik-hegde]

I spent quite some time debugging the NaN issue in my system and finally realized this is the issue. For ex: If you have 101 classes and you encode labels from 1-101, your model will always have a NaN! Because 101 is greater than the size of the logits!

Can we have at least a Warning popping up, instead of simply showing that the loss diverged with a NaN? That can bring down the debugging time to a great extent for many users.

[Sayan-Paul]

I spent quite some time debugging the NaN issue in my system and finally realized this is the issue. For ex: If you have 101 classes and you encode labels from 1-101, your model will always have a NaN! Because 101 is greater than the size of the logits!

Can we have at least a Warning popping up, instead of simply showing that the loss diverged with a NaN? That can bring down the debugging time to a great extent for many users.

Finally, the error makes sense. Spent a day printing tensors and trying to debug.

[ebrevdo]

This is reasonable to add but nontrivial. Rasmus what's a good base example for returning errors from the GPU?

…

On Fri, Oct 19, 2018, 6:29 PM Sayan Paul ***@***.*** wrote: I spent quite some time debugging the NaN issue in my system and finally realized this is the issue. For ex: If you have 101 classes and you encode labels from 1-101, your model will always have a NaN! Because 101 is greater than the size of the logits! Can we have at least a Warning popping up, instead of simply showing that the loss diverged with a NaN? That can bring down the debugging time to a great extent for many users. Finally, the error makes sense. Spent a day printing tensors and trying to debug. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#8484 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABtim0C4V0BCH2SYpjlQpmhbekDtVElDks5umnyDgaJpZM4MgDu4> .

[gilgarad]

@karthik-hegde you saved me. The problem I had is related to what you mentioned. I have a few differences between the number of categories I assigned and the actual number.

[tchaye59]

I spent quite some time debugging the NaN issue in my system and finally realized this is the issue. For ex: If you have 101 classes and you encode labels from 1-101, your model will always have a NaN! Because 101 is greater than the size of the logits!

Can we have at least a Warning popping up, instead of simply showing that the loss diverged with a NaN? That can bring down the debugging time to a great extent for many users.

You just save my day. Thanks a lot. My target vector is a pad with zeros and I forget to consider the 0 class in the final layer

[desmanda]

I am really disappointed with TF, I lost a lot of time just debugging the nan in my loss, thinking of all the possibilities, and then it turns out it is just a wrong error message! Thanks, everyone for opening this issue, you saved me!