FusedBatchNorm & Conv2D backwards doesn't support zero batch size

[ppwwyyxx]

Most ops in TF work well with tensors with zero elements. However, convolution fusedbatchnorm with cudnn gives the following error:

2017-11-17 08:00:20.835113: F tensorflow/stream_executor/cuda/cuda_dnn.cc:444] could not convert BatchDescriptor {count: 0 feature_map_count: 1024 spatial: 28 28  value_min: 0.000000 value_max: 0.000000 layout: BatchDepthYX} to cudnn tensor descriptor: CUDNN_STATUS_BAD_PARAM 

I would expect it checks and returns a 4D tensor with zero batch-size. Currently I have to work around it by tf.cond.

[cy89]

@zheng-xq this seems like it falls between the boundaries of cuDNN and TF. Is this something we might fix, or should I set to "contributions welcome"?

[facaiy]

Perhaps doc of cudnnSetTensorNdDescriptor is helpful for discussion.

Return Value	Meaning
CUDNN_STATUS_BAD_PARAM	At least one of the elements of the array dimA was negative or zero, or dataType has an invalid enumerant value

[hpnhxxwn]

@ppwwyyxx
Hi, I'm facing the same issue using CUDA 8.0 with cuDNN 6. Could you show me how did you use tf.cond? Like which parameter did you put condition on?

Thank you!

[ppwwyyxx]

https://github.com/ppwwyyxx/tensorpack/blob/be50085f7e2d9ed2704a14cb4c2e6aaa20548972/examples/FasterRCNN/train.py#L123-L134

[hpnhxxwn]

@ppwwyyxx Could you point me how to use tf.cond on keras generator type of input?

[ppwwyyxx]

I'm not sure I know what you mean. And I think it's already unrelated to the issue anyway. You can search for documentation/examples or ask on stackoverflow for usage questions.

[ppwwyyxx]

Turned out that Conv2d actually does support empty tensor, but FusedBatchNorm does not. That is probably why I saw this error. To reproduce:

with tf.device('/gpu:0'):
    x = tf.random_normal([0, 16, 64, 64])
    scale = tf.random_normal([16])
    y = tf.nn.fused_batch_norm(x, scale, scale, data_format='NCHW')

with tf.Session() as sess:
    print(sess.run(y))

[ppwwyyxx]

There is a problem with conv2d backward as well, though forward works:

with tf.device('/gpu:0'):
    x = tf.random_normal([0, 16, 64, 64])
    W = tf.random_normal([3, 3, 16, 16])
    y = tf.nn.conv2d(x, W, [1, 1, 1, 1], padding='SAME', data_format='NCHW')
    grad = tf.gradients(y, W)


with tf.Session() as sess:
    print(sess.run(grad))

[ppwwyyxx]

Anyone can take a look at my PR #15264?

[yzhwang]

Thanks for the PR @ppwwyyxx . I will take a look.