Using AutoGrad with CuArray

[ngphuoc]

I modified the housing example to use CuArray as shown below. The forward pass is OK but in the backward pass it causes ERROR: MethodError: no method matching next(::AutoGrad.Rec{CuArray{Float32,2}}, ::Tuple{Base.OneTo{Int64},Int64})

using Knet,CuArrays
include(Knet.dir("data","housing.jl"))
data = housing()
x,y = data
w = Any[ 0.1f0*cu(randn(Float32,1,13)), 0.0f0 ]

predict(w,x) = w[1]*x .+ w[2]

loss(w,x,y) = mean(abs2,y-predict(w,x))
loss(w,x,y)  # 593.6816f0

lossgradient = grad(loss)
lossgradient(w,x,y)  # Error: MethodError: no method matching next(::AutoGrad.Rec{CuArray{Float32,2}}, ::Tuple{Base.OneTo{Int64},Int64})

function train(w, data; lr=.1)
  for d=data
    x,y = cu.(d)
    dw = lossgradient(w, x, y)
    for i in 1:length(w)
      w[i] -= lr * dw[i]
    end
  end
  return w
end


for i=1:10; train(w, [data]); println(loss(w,x,y)); end

[ngphuoc]

I have checked that AutoGrad can work with ArrayFire: JuliaGPU/ArrayFire.jl#183

PS: if my benchmark was correct then KnetArray is about 3-4 times faster.

[CarloLucibello]

autograd gradient is working now (julia 0.7 and CuArrays master) but it is of the wrong type (standard array)

julia> using Knet,CuArrays, Statistics

julia> x,y = rand(13,10), randn(1,10)
([0.774427 0.275647 … 0.452615 0.09355; 0.411376 0.765637 … 0.527263 0.839156; … ; 0.650629 0.404082 … 0.520125 0.279594; 0.743382 0.192372 … 0.756798 0.457104], [-0.816243 -0.606755 … 1.3648 -0.116751])

julia> w = Any[ 0.1f0*cu(randn(Float32,1,13)), 0.0f0 ]
2-element Array{Any,1}:
  Float32[-0.092651 -0.0714096 … -0.102493 0.080609]
 0.0f0                                              

julia> w[1] |> typeof
CuArray{Float32,2}

julia> predict(w,x) = w[1]*x .+ w[2]
predict (generic function with 1 method)

julia> loss(w,x,y) = mean(abs2, y.- predict(w,x))
loss (generic function with 1 method)

julia> loss(w,x,y)
0.7325428950542855

julia> grad(loss)(w,x,y)  
2-element Array{Any,1}:
  [0.215699 0.272719 … 0.196543 0.123026]
 0.4017888307995154                      

julia> grad(loss)(w,x,y)[1] |> typeof
Array{Float64,2}