make use of conv_bias_act

[gartangh]

Make use of conv_bias_act from FluxML/NNlib.jl#228.
Should speed up Convolutional Neural Networks significantly after JuliaGPU/CUDA.jl#321.

[DhairyaLGandhi]

Bump

[DhairyaLGandhi]

bors try

[bors]

try

Build failed:

• ci/gitlab/gitlab.com

[DhairyaLGandhi]

Bump

[DhairyaLGandhi]

Suggested change

	conv_bias_act(x, w, cdims::DenseConvDims, b::Zeros, σ) = σ.(conv(x, w, cdims))
	function conv_bias_act(x::CuArray, w::CuArray{T}, cdims::DenseConvDims, b::Zeros, σ) where T
	bz = CUDA.zeros(size(b)...)
	NNlib.conv_bias_act(x, w, cdims, bz, σ)
	end

[DhairyaLGandhi]

We need this materialised here since CUDNN expects to get an array here, and the bump in performance over regular Conv calls is enough to justify needing to allocate some memory and quickly putting it back into the pool. We need them 2x speedups on ResNets!

[DhairyaLGandhi]

Use reinterpret(reshape, ...) instead

[CarloLucibello]

using the conv_bias_act is an inner implementation detail that should not block v0.12 (especially given the fact that the author is unresponsive)

[DhairyaLGandhi]

Its not an inner detail - conv being equivalent to the forward pass of Conv is essentially the functional form of the layer. Changing that out for a different function is a breaking change since conv and conv_bias_act aren't compatible functions. (conv does not involve about the bias term for example. conv_bias_act also goes through a completely different code path with CUDA with different assumptions regarding accuracy and the algorithms that can be chosen for the convolution)

[CarloLucibello]

Its not an inner detail - conv being equivalent to the forward pass of Conv is essentially the functional form of the layer.

conv is not equivalent to the forward pass of Conv, Conv also adds a bias an activation on top of that, which means that Conv is equivalent to conv_bias_act + keeping an inner state

[ToucheSir]

conv_bias_act also goes through a completely different code path with CUDA with different assumptions regarding accuracy and the algorithms that can be chosen for the convolution)

In that case, we should document these assumptions and the tolerences we're okay with in different scenarios. e.g. is Conv allowed to switch to conv_bias_act when bias is set and we're running in a @fastmath context? Whatever the solution, I'm strongly of the opinion that a) we should make conv_bias_act easier to use, and b) defining a separate layer for it is a bad idea.

[DhairyaLGandhi]

Well, composition means that we don't need to have wrappers around kernels that effectively compute the raison d'etre for the layer without polluting the namespace/ api. The api to get all the details of a layer for a "function" would be pretty ugly. Its not a big deal for users to broadcast an activation over a conv either.

[ToucheSir]

AIUI, the whole point of conv_bias_act is that certain runtimes (e.g. CuDNN) will provide accelerated (e.g. fused) versions that run faster than the usual act(conv(W, x) .+ b) (which would incur at least 3 dispatches/launches). There may be marginally different results because of how different implementations work, but that's why I mentioned having something like @fastmath.

If we don't make use of conv_bias_act in Flux itself, I can count on one hand the number of people who would use it. There's very little harm in having it as a fast path of Conv that only triggers under specific circumstances.

[mcabbott]

Seems sensible for the fast way to be the default, no? Flux surely doesn't guarantee that exact floating-point values will be the same between versions, and it's definitely not a library targeting uses which care a lot about the 16th decimal place.

I don't see any discussion of how different this is; if it's more than minor floating point stuff can someone summarise & provide links? If it is e.g. substantially less accurate, then you could argue for some Conv(; precise=true) or some global switch or something.

[DhairyaLGandhi]

some global switch or something.

CUDA.math_mode, and fastmath etc