Eagerly put objects back in the pool

[maleadt]

Basically, something like #226, but more careful and with actual uses in CuArrays.jl. Some backstory.

We now depend on the Julia GC to collect CuArray objects before they end up in the pool again. This is not optimal, and compounded by the fact that the Julia GC doens't know about the GPU's memory pressure. As a result, we kind-of rely on hitting a GPU OOM before collecting dead arrays and putting them back into the pool:

CuArrays.jl/src/memory.jl

Lines 254 to 256 in 8525a9d

	ex == CUDAdrv.ERROR_OUT_OF_MEMORY || rethrow()
	# 3. that failed; make Julia collect objects and check 1. again
	gc(true) # full collection

On the other hand, we often know that arrays are unused and could be put back into the pool again. For example:

CuArrays.jl/src/solver/dense.jl

Lines 126 to 138 in 0384d25

	buffer = CuArray{$elty}(undef, bufSize[])
	tau = CuArray{$elty}(undef, min(m, n))
	devinfo = CuArray{Cint}(undef, 1)
	@check ccall(($(string(fname)),libcusolver), cusolverStatus_t,
	(cusolverDnHandle_t, Cint, Cint, Ptr{$elty},
	Cint, Ptr{$elty}, Ptr{$elty}, Cint, Ptr{Cint}),
	dense_handle(), m, n, A, lda, tau, buffer,
	bufSize[], devinfo)
	info = _getindex(devinfo, 1)
	if info < 0
	throw(ArgumentError("The $(info)th parameter is wrong"))
	end
	A, tau

One option is to call finalize, but that is a pretty slow call and shouldn't be used in performance critical code. Instead, we should have a call to eagerly "free" objects (from the perspective of CuArrays.jl) and put them back into the pool. For example, the geqrf! method above would then free its buffer before returning. To prevent use of early-freed arrays, we should probably wipe the memory buffer handle (eg. set to NULL or some other sentinel value). Care should be taken to deal with ownership (not support this call when we don't own the CuArray? make sure the CUDAdrv refcount is 1?).

[maleadt]

cc @SuperFluffy -- would be good to have a use-case or benchmark.

[KristofferC]

The common way of doing this is (of course) with the do syntax, but this will likely make the code a lot uglier and more "indent"-y.

[maleadt]

Implemented both.