Add @autosize

[mcabbott]

The macro replaces _ with an inferred input size:

julia> @autosize (2,3,4) Chain(Dense(_ => 5, tanh), Flux.flatten, Maxout(() -> Dense(_ => 3), 2))
Chain(
  Dense(2 => 5, tanh),                  # 15 parameters
  Flux.flatten,
  Maxout(
    Dense(15 => 3),                     # 48 parameters
    Dense(15 => 3),                     # 48 parameters
  ),
)                   # Total: 6 arrays, 111 parameters, 892 bytes.

julia> ans(rand(2,3,4)) |> size
(3, 4)

PR Checklist

• Tests are added
• Entry in NEWS.md
• Documentation, if applicable

[ToucheSir]

Could we force users to call recursive_striplazy(model, input_size) or something before using an incrementally constructed network like this? Maybe define a rrule which throws an error?

[mcabbott]

striplazy should be fully recursive. We could make a function that calls this after outputsize & returns the model. And indeed an rrule would be one way to forbid you not to strip the model before using it for real.

I suppose the other policy would just be to allow these things to survive in the model. As long as you never change it, and don't care about the cost of the if & type instability, it should work?

But any use outside of @autosize probably needs another macro... writing Flux.LazyLayer("", x -> Dense(size(x,1) => 10), nothing) seems sufficiently obscure that perhaps it's OK to say that's obviously at own risk, for now? @autosize can be the only API until we decide if we want more.

[ToucheSir]

@mcabbott I missed this in the review, but GPU tests are failing because the |> here is binding more tightly than @autosize and thus the model isn't actually moved onto the GPU. Is there anything we can do about that other than adding more parens? Whatever changes made would affect the @autosize docstring as well, since it shares this example.

[mcabbott]

Oh no I thought I checked this, sorry. It's not working but the binding is what I expected:

julia> (@autosize (1,2) Dense(_, 3) |> f64).bias
3-element Vector{Float32}:
 0.0
 0.0
 0.0

julia> :(@autosize (1,2) Dense(_, 3) |> f64) |> dump
Expr
  head: Symbol macrocall
  args: Array{Any}((4,))
    1: Symbol @autosize
    2: LineNumberNode
      line: Int64 1
      file: Symbol REPL[16]
    3: Expr
      head: Symbol tuple
      args: Array{Any}((2,))
        1: Int64 1
        2: Int64 2
    4: Expr
      head: Symbol call
      args: Array{Any}((3,))
        1: Symbol |>
        2: Expr
          head: Symbol call
          args: Array{Any}((3,))
            1: Symbol Dense
            2: Symbol _
            3: Int64 3
        3: Symbol f64

It seems the gpu walk is taking place too early. Maybe I never implemented my scheme to delay it.

[mcabbott]

My scheme was I think aimed at adapt. There you can grab the function being mapped, and replace the layer's maker function with one composed with this. But for Functors.jl I think that's impossible.

So we should just make it an error. And remove this use from the docs. Call gpu once it's done, i.e. with brackets.

[ToucheSir]

Sounds good to me.