Added support for LSTM and BatchNorm #24

Roboneet · 2018-06-24T11:19:45Z

StagedArray uses val field instead of dims.

MikeInnes · 2018-07-04T14:05:53Z

lib/flux.js

+const add = (a, b) => a + b;
+const sub = (a, b) => a - b;
+const mul = (a, b) => a * b;
+const div = (a, b) => a / b;


Is it possible to just emit these directly?

MikeInnes · 2018-07-04T14:09:36Z

Looks good to me, though I guess it needs rebasing over the NCHW changes.

MikeInnes · 2018-07-04T14:10:16Z

It would also be good to start having some basic tests for the generated code here.

MikeInnes · 2018-07-12T15:32:15Z

Looks like this has conflicts from the tensorflow.js update

MikeInnes · 2018-07-16T15:41:55Z

👍 Thanks!

amellnik · 2018-07-18T03:26:00Z

I'm late to the party, but I'm still seeing Unsupported type Tuple{TrackedArray{…,Array{Float64,1}},TrackedArray{…,Array{Float64,1}}} when trying to export a model that follows the char-rnn example from the model zoo. Is there something I'm missing?

Roboneet · 2018-07-18T07:50:11Z

Can you please post the code? I can't seem to reproduce the error with the char-rnn model.

amellnik · 2018-07-18T15:13:40Z

Here's a quick but not very minimal example:

using Flux
using Flux: onehot, argmax, chunk, batchseq, throttle, crossentropy
using FluxJS
using StatsBase: wsample
using Base.Iterators: partition
using BSON: @save, @load

text = collect(readstring("hp.txt")) # Or some other text file
alphabet = [unique(text)..., '_']
text = map(ch -> onehot(ch, alphabet), text)
stop = onehot('_', alphabet)
N = length(alphabet)

m = Chain(
  LSTM(N, 128),
  LSTM(128, 128),
  Dense(128, N),
  softmax)

m(trues(94)) # Returns a Tracked 94-element Array{Float64,1}:

@code_js m(trues(94)) # Unsupported type Tuple{TrackedArray{…,Array{Float64,1}},TrackedArray{…,Array{Float64,1}}}

I get this error no matter what I call m with in @code_js.

Roboneet · 2018-07-18T15:44:47Z

That error seems to come up on commits before 4afb9e7. Can you check the git logs and see if you have the latest version of master?

amellnik · 2018-07-19T04:54:39Z

Ugh, my mistake -- thanks. On actual master I am still getting the following:

typeof(f) = Base.#one
one
Tuple{TrackedArray{…,Array{Float64,1}}}
no method found for the specified argument types

Roboneet · 2018-07-21T12:52:58Z

I'm not sure why this is happening, I can't reproduce it on my end. Now that we have tests for the primitives, can you check out which ones aren't working on yours?

amellnik · 2018-07-22T00:28:57Z

I was using the ASTInterpreter2 master which was also causing Flux tests to fail. I freed it back to 0.1.1, and while Flux tests now pass, FluxJS tests (and exporting models) fails with the following:

FluxJS: Error During Test
  Got an exception of type UndefRefError outside of a @test
  UndefRefError: access to undefined reference
  Stacktrace:
   [1] lookup_var_if_var at /home/alex/.julia/v0.6/ASTInterpreter2/src/ASTInterpreter2.jl:147 [inlined]
   [2] lookup(::ASTInterpreter2.JuliaStackFrame, ::SSAValue) at /home/alex/.julia/v0.6/Vinyl/src/interpret.jl:21
   [3] broadcast_t(::Function, ::Type{Any}, ::Tuple{Base.OneTo{Int64}}, ::CartesianRange{CartesianIndex{1}}, ::ASTInterpreter2.JuliaStackFrame, ::Array{Any,1}) at ./broadcast.jl:258
   [4] broadcast_c at ./broadcast.jl:321 [inlined]
   [5] broadcast(::Function, ::ASTInterpreter2.JuliaStackFrame, ::Array{Any,1}) at ./broadcast.jl:455
   [6] callargs(::DebuggerFramework.DebuggerState) at /home/alex/.julia/v0.6/Vinyl/src/interpret.jl:29
   [7] runall(::FluxJS.BTrace, ::DebuggerFramework.DebuggerState) at /home/alex/.julia/v0.6/Vinyl/src/interpret.jl:34
   [8] overdub(::FluxJS.BTrace, ::Function) at /home/alex/.julia/v0.6/Vinyl/src/interpret.jl:53
   [9] primitive(::FluxJS.Trace, ::Base.#broadcast, ::Function, ::Tuple{TrackedArray{…,Array{Float64,1}}}) at /home/alex/.julia/v0.6/Vinyl/src/hooks.jl:24
   [10] runall(::FluxJS.Trace, ::DebuggerFramework.DebuggerState) at /home/alex/.julia/v0.6/Vinyl/src/interpret.jl:37
   [11] overdub(::FluxJS.Trace, ::Flux.Dense{Base.#identity,TrackedArray{…,Array{Float64,2}},TrackedArray{…,Array{Float64,1}}}, ::FluxJS.StagedArray{Float64,1}, ::Vararg{FluxJS.StagedArray{Float64,1},N} where N) at /home/alex/.julia/v0.6/Vinyl/src/interpret.jl:53
   [12] #trace#7(::FluxJS.Trace, ::Function, ::Flux.Dense{Base.#identity,TrackedArray{…,Array{Float64,2}},TrackedArray{…,Array{Float64,1}}}, ::FluxJS.StagedArray{Float64,1}, ::Vararg{FluxJS.StagedArray{Float64,1},N} where N) at /home/alex/.julia/v0.6/FluxJS/src/trace.jl:46
   [13] (::FluxJS.#kw##trace)(::Array{Any,1}, ::FluxJS.#trace, ::Flux.Dense{Base.#identity,TrackedArray{…,Array{Float64,2}},TrackedArray{…,Array{Float64,1}}}, ::FluxJS.StagedArray{Float64,1}, ::Vararg{FluxJS.StagedArray{Float64,1},N} where N) at ./<missing>:0
   [14] #_traceλ#10(::FluxJS.Trace, ::Function, ::Flux.Dense{Base.#identity,TrackedArray{…,Array{Float64,2}},TrackedArray{…,Array{Float64,1}}}, ::Array{Float64,1}, ::Vararg{Array{Float64,1},N} where N) at /home/alex/.julia/v0.6/FluxJS/src/trace.jl:55
   [15] (::FluxJS.#kw##_traceλ)(::Array{Any,1}, ::FluxJS.#_traceλ, ::Flux.Dense{Base.#identity,TrackedArray{…,Array{Float64,2}},TrackedArray{…,Array{Float64,1}}}, ::Array{Float64,1}, ::Vararg{Array{Float64,1},N} where N) at ./<missing>:0
   [16] #traceλ#11 at /home/alex/.julia/v0.6/FluxJS/src/trace.jl:60 [inlined]
   [17] traceλ at /home/alex/.julia/v0.6/FluxJS/src/trace.jl:60 [inlined]
   [18] macro expansion at /home/alex/.julia/v0.6/FluxJS/test/runtests.jl:12 [inlined]
   [19] macro expansion at ./test.jl:860 [inlined]
   [20] anonymous at ./<missing>:?
   [21] include_from_node1(::String) at ./loading.jl:576
   [22] include(::String) at ./sysimg.jl:14
   [23] process_options(::Base.JLOptions) at ./client.jl:305
   [24] _start() at ./client.jl:371
Test Summary: | Error  Total
FluxJS        |     1      1
ERROR: LoadError: Some tests did not pass: 0 passed, 0 failed, 1 errored, 0 broken.
while loading /home/alex/.julia/v0.6/FluxJS/test/runtests.jl, in expression starting on line 9
FluxJS had test errors

Stacktrace:
 [1] #test#62(::Bool, ::Function, ::Array{AbstractString,1}) at ./pkg/entry.jl:759
 [2] (::Base.Pkg.Entry.#kw##test)(::Array{Any,1}, ::Base.Pkg.Entry.#test, ::Array{AbstractString,1}) at ./<missing>:0
 [3] (::Base.Pkg.Dir.##4#7{Array{Any,1},Base.Pkg.Entry.#test,Tuple{Array{AbstractString,1}}})() at ./pkg/dir.jl:36
 [4] cd(::Base.Pkg.Dir.##4#7{Array{Any,1},Base.Pkg.Entry.#test,Tuple{Array{AbstractString,1}}}, ::String) at ./file.jl:70
 [5] #cd#1(::Array{Any,1}, ::Function, ::Function, ::Array{AbstractString,1}, ::Vararg{Array{AbstractString,1},N} where N) at ./pkg/dir.jl:36
 [6] (::Base.Pkg.Dir.#kw##cd)(::Array{Any,1}, ::Base.Pkg.Dir.#cd, ::Function, ::Array{AbstractString,1}, ::Vararg{Array{AbstractString,1},N} where N) at ./<missing>:0
 [7] #test#3(::Bool, ::Function, ::String, ::Vararg{String,N} where N) at ./pkg/pkg.jl:276
 [8] test(::String, ::Vararg{String,N} where N) at ./pkg/pkg.jl:276

What ASTInterpreter2 are you on?

Roboneet · 2018-07-22T09:14:15Z

I'm on master as well. You might want to try out this branch of Vinyl to get a stacktrace of the error.

amellnik · 2018-07-30T02:42:24Z

UndefRefError: access to undefined reference
Stacktrace of evaluated expression:
[1] #23() at :0

Stacktrace of evaluated expression:
[1] LSTMCell(h_, x) at /home/alex/.julia/v0.6/Flux/src/layers/recurrent.jl:132

Stacktrace of evaluated expression:
[1] #82(x, m) at /home/alex/.julia/v0.6/Flux/src/layers/basic.jl:31

Stacktrace of evaluated expression:
[1] mapfoldl_impl(f, op, v0, itr, i) at reduce.jl:39

Stacktrace of evaluated expression:
[1] mapfoldl(f, op, v0, itr) at reduce.jl:58

Stacktrace of evaluated expression:
[1] foldl(op, v0, itr) at reduce.jl:87

Stacktrace of evaluated expression:
[1] Chain(x) at /home/alex/.julia/v0.6/Flux/src/layers/basic.jl:31

Roboneet added 13 commits June 16, 2018 15:04

size issue

274f2d3

LSTM

4afb9e7

0 based index

7441e1f

batchnorm

5eb8692

batchnorm support

4013a71

reshape colon

235c6a9

+ and copy

b9f5dac

io

ee7fa75

tensor

02aff05

String prop names

24ad033

get prop

8853934

get states

f15526d

mean

50c334b

MikeInnes reviewed Jul 4, 2018

View reviewed changes

Roboneet added 2 commits July 4, 2018 23:19

removed extra functions

eb1f2b9

Merge branch 'master' into dev_2

ae424f3

Roboneet added 2 commits July 16, 2018 00:26

resolve conflicts

46e82c1

batchnorm fix and more primitives

7af27a2

MikeInnes merged commit c2eb06e into FluxML:master Jul 16, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added support for LSTM and BatchNorm #24

Added support for LSTM and BatchNorm #24

Roboneet commented Jun 24, 2018

MikeInnes Jul 4, 2018

MikeInnes commented Jul 4, 2018

MikeInnes commented Jul 4, 2018

MikeInnes commented Jul 12, 2018

MikeInnes commented Jul 16, 2018

amellnik commented Jul 18, 2018

Roboneet commented Jul 18, 2018

amellnik commented Jul 18, 2018 •

edited

Roboneet commented Jul 18, 2018

amellnik commented Jul 19, 2018

Roboneet commented Jul 21, 2018

amellnik commented Jul 22, 2018

Roboneet commented Jul 22, 2018

amellnik commented Jul 30, 2018

Added support for LSTM and BatchNorm #24

Added support for LSTM and BatchNorm #24

Conversation

Roboneet commented Jun 24, 2018

MikeInnes Jul 4, 2018

Choose a reason for hiding this comment

MikeInnes commented Jul 4, 2018

MikeInnes commented Jul 4, 2018

MikeInnes commented Jul 12, 2018

MikeInnes commented Jul 16, 2018

amellnik commented Jul 18, 2018

Roboneet commented Jul 18, 2018

amellnik commented Jul 18, 2018 • edited

Roboneet commented Jul 18, 2018

amellnik commented Jul 19, 2018

Roboneet commented Jul 21, 2018

amellnik commented Jul 22, 2018

Roboneet commented Jul 22, 2018

amellnik commented Jul 30, 2018

amellnik commented Jul 18, 2018 •

edited