[feature request] disable harmful broadcast

[raingo]

Broadcast may lead to hidden bugs. For example, the following code,

xent = tf.nn. sigmoid_cross_entropy_with_logits(logits, y)
loss = tf.reduce_mean(xent)

When logits is of the shape (200,1) and the y is of the shape (200,), the xent becomes (200, 200), which is wrong. What's worse, the reduce_mean will hide this problem by reducing everything into a scalar.

Request,

1. more shape validation in sigmoid_cross_entropy_with_logits
2. option to totally disable broadcast

[vrv]

1. is done:

   tensorflow/tensorflow/python/ops/nn.py

   Line 287 in 3871973

   raise ValueError(

   -- will be in our next binary release.

[raingo]

Thanks. I guess the second is impossible. so closing.

[vrv]

Nothing is impossible, it would just take a lot of work to plumb that through everywhere, to set the option globally, and for everything to work as well without broadcasting. For example, when computing gradients, automatic broadcasting is quite useful to simplify gradient expressions, particularly when the shapes are unknown at construction time. I've seen similar requests made for numpy, and it sounds like higher level libraries typically provide the 'safety' for those users who want it. However, I'm open to the idea if it can be done in a nice way.

[rasmusbergpalm]

I have spent 8 hours on exactly this bug. An option to disable broadcasting globally or raising a warning, etc. would be very nice.

[agentydragon]

Just ran into a bug which would also have been prevented by warning on broadcasting. A a * b operation ended up doing vector * vector -> matrix multiplication instead of vector * vector -> vector elementwise multiplication. Cost me on the order of tens of hours (writeup). Would be nice to either have the option for such a warning, or at least more specific operators that require the programmer to be more explicit about intent. (e.g.: tf.math.elementwise_multiply).