Avoids running unneeded backprop kernels.#5902
Conversation
src/operator/cudnn_convolution-inl.h
Outdated
| typename DataType<DType>::ScaleType beta = 0.0f; | ||
| typename DataType<DType>::ScaleType beta_add = 1.0f; | ||
| if (!param_.no_bias) { | ||
| CHECK_NE(req[conv::kBias], kNullOp); |
There was a problem hiding this comment.
why check instead of if?
There was a problem hiding this comment.
This line was introduced to mirror the handling seen in cudnn_deconvolution-inl.h. To summarize the behavior as coded here for the 4 possible cases:
param_.no_bias == true, req[conv::kBias] == kNullOp - avoids backward-to-bias kernel
param_.no_bias == true, req[conv::kBias] != kNullOp - avoids backward-to-bias kernel
param_.no_bias == false, req[conv::kBias] == kNullOp - throws exception
param_.no_bias == false, req[conv::kBias] != kNullOp - runs backward-to-bias kernel
Would you prefer a different behavior?
There was a problem hiding this comment.
I think it should be
if (!param_.no_bias && req[conv::kBias] != kNullOp)
* Avoids running unneeded backprop kernels. * Permits convolution bias that is not backpropped to.
2 participants
Within CuDNNConvolutionOp.Backward(), this adds checks to avoid launching backprop kernels unnecessarily (i.e. when req[] == kNullOp). This would result typically in avoiding the L1 dgrad kernel launch in CNN's.