gradcheck fails for broadcast operation

[ekinakyurek]

I came across with interesting bug in gradcheck. When w is initialized with zeros gradcheck fails. Otherwise it gives the correct output.

julia> using Knet
julia> w = KnetArray(zeros(1,1024));
julia> x = KnetArray(randn(14,1024));
julia> broadcasting(w,x) =  mean(x .+ w);
julia> gradcheck(broadcasting,w,x;verbose=true)

WARNING: d=0.0009765625 nd=0.0
WARNING: d=0.0009765625 nd=0.0
WARNING: d=0.0009765625 nd=0.0
WARNING: d=0.0009765625 nd=0.0
WARNING: d=0.0009765625 nd=0.0
WARNING: d=0.0009765625 nd=0.0
WARNING: d=0.0009765625 nd=0.0
WARNING: d=0.0009765625 nd=0.0
WARNING: d=0.0009765625 nd=0.0
WARNING: d=0.0009765625 nd=0.0
false

julia> w = KnetArray(randn(1,1024));
julia> gradcheck(broadcasting,w,x;verbose=true)
gcheck: d=0.0009765625 nd=0.0009765625004297092
gcheck: d=0.0009765625 nd=0.0009765624997314316
gcheck: d=0.0009765625 nd=0.0009765625004655185
gcheck: d=0.0009765625 nd=0.0009765624997135271
gcheck: d=0.0009765625 nd=0.000976562500447614
gcheck: d=0.0009765625 nd=0.000976562500165617
gcheck: d=0.0009765625 nd=0.0009765624997135271
gcheck: d=0.0009765625 nd=0.0009765624997135271
gcheck: d=0.0009765625 nd=0.0009765625004526496
gcheck: d=0.0009765625 nd=0.0009765624997493363
true

[CarloLucibello]

This is due to limited numerical precision, since the elements of w=zeros(..) in the numerical derivative are evaluated at the very small numbers

w1 = -8.515919680016301e-109
w2 = 8.515919680016301e-109

due to the default gc_dx(x::Number)=cbrt(eps(float(x))), and therefore they are ignored when summed with the element of x, resulting numerically in f(w2) == f(w1).

By the way, the gradcheck method now is not even in the library but in test/gradcheck.jl

[denizyuret]

@CarloLucibello is correct: in the output above, nd is the numerical derivative and d is the derivative computed by the program. Even though the numerical derivative is false due to the numeric error, grad computes d=1/1024 which is the correct result.

I am moving gradcheck, addtest, display and other non-essential functions out of the package to reduce bloat. They are still available as stand-alone utilities in AutoGrad.dir("util") and AutoGrad.dir("test").