fvec

[mcabbott]

This is an attempt to write a function which does what Flux.destructure does, but more carefully. It allows for missing branches of the structural gradient, and for shared weights having separate gradients, which ought to accumulate in making a vector. When returning to a structural gradient, only the first node gets the gradient, to avoid mistakenly doubling it.

It also declares some Base types like Transpose to be functors, since a decoder with transposed weights seems like the canonical example. And sets up a way to allow their gradients not to be Transpose.

And, thirdly, it changes fmap to not use the cache on isbits objects. This is to avoid falsely tying two parameters initially say SA[0,0,0]. We thought elsewhere that the way to indicate that you do what such things tied is to wrap them in some TiedArray type like https://gist.github.com/mcabbott/35f660ff9fb8e7b0b23a2abb94618cf4, but not in this PR.

---

Most of this can simply run with a different walk to hit only trainable arrays. Which I think ought to be done in Optimisers.jl, which owns trainable. The exception is that the gradient of fcopy doesn't seem to fit into just using a different walk, maybe there's a way? It takes a different children function instead.

Not sure what should be exported here, or what anything should be called. I think the name destructure should belong to Optimisers.jl too. Perhaps this package should export a function which similarly returns a vector and a reconstructor. That's just one line after the pieces defined here. (But easier to write & test the pieces separately.)

---

Needs more tests, still.

[mcabbott]

This is #28 and should perhaps be pulled out.

From FluxML/Optimisers.jl#42 I think the right pattern is actually more like this -- calling parent(x) is not safe, and we may want to transpose things which aren't matrix-like.

Functors.functor(::Type{<:Transpose}, x) = (parent = _transpose(x),), y -> _transpose(only(y))
_transpose(x) = transpose(x)
_transpose(x::NamedTuple{(:parent,)}) = x.parent  # "structural" gradient
# _transpose(bc::Broadcast.Broadcasted{S}) where S = Broadcast.Broadcasted{S}(map(_transpose, bc.args)) do args...
#     transpose(bc.f(map(transpose, args)...))
# end
_transpose(bc::Broadcast.Broadcasted) = transpose(Broadcast.materialize(bc))