Introduce Composite for structured deriviatives

[oxinabox]

This will finally finish resolving #8

I think before it is done it needs unthunk to be used instead of extern #56

[nickrobinson251]

I am enjoying following along :)

[oxinabox]

Spoke to @MikeInnes and co. in the ML call last week.
If an inner constructor is not provided,
we should not bypass the lack,
because it may encode an invarient.

So we ask the user what to do, they can either provide a default contructor,
overload ChainRulesCore.construct(::Type{Primal}, ::NamedTuple)
or overload +(::Primal, ::Composite{Primal})

Except for closures.
Closures have no constructors, for performance reasons.
Jeff says it is fine to construct them via the same way Serialization does.
but we can add them in a follow up PR.
The Closure case is not hugely interesting for us, since it is pretty weird to have a handle on one to be able to define its deriviative

[nickrobinson251]

took a quick look and seems grand. excited to see this happening. thanks a lot and sorry i don't have more time to review

[oxinabox]

sorry i don't have more time to review

You reviewed this at 8pm.

[willtebbutt]

Broadly very nice. It would be nice to see some unit tests for the various bits and bobs in src/composite_core.jl in a corresponding file in test, as opposed to just integration tests via test/differentials.jl. Also, some tests with nested composites probably wouldn't be a bad idea.

[oxinabox]

I put some effort to use colors to make it a good log message when you do addition and it can't make a primal.

In this example StructWithInvarient is the type of value, so the user should know what that is

rather than that all being in red

[mattBrzezinski]

Looks good to me :)

[willtebbutt]

Trying to use Composite to work with NamedTuples:

A = (a=rand(5), b=rand(5, 5))
dA = Composite{typeof(A)}(;A...)
2dA # this works :)
A + dA # this errors :(

the last line yields

julia> A + dA
ERROR: Could not construct NamedTuple{(:a, :b),Tuple{Array{Float64,1},Array{Float64,2}}} after addition.
This probably means no default constructor is defined.
Either define a default constructor
NamedTuple{(:a, :b),Tuple{Array{Float64,1},Array{Float64,2}}}(1, 2)
or overload
ChainRulesCore.construct(::Type{NamedTuple{(:a, :b),Tuple{Array{Float64,1},Array{Float64,2}}}}, ::Composite{NamedTuple{(:a, :b),Tuple{Array{Float64,1},Array{Float64,2}}},Tuple{Array{Float64,1},Array{Float64,2}}})
or overload
Base.:+(::NamedTuple{(:a, :b),Tuple{Array{Float64,1},Array{Float64,2}}}, ::Composite{NamedTuple{(:a, :b),Tuple{Array{Float64,1},Array{Float64,2}}},Tuple{Array{Float64,1},Array{Float64,2}}})
Original Exception:
MethodError(ChainRulesCore.elementwise_add, ((a = [0.6656951461385991, 0.4645935051577579], b = [0.13576750972690776 0.19906834130647622; 0.22171335040027196 0.6481124619162637]), ([0.6656951461385991, 0.4645935051577579], [0.13576750972690776 0.19906834130647622; 0.22171335040027196 0.6481124619162637])), 0x0000000000006886)

Stacktrace:
 [1] +(::NamedTuple{(:a, :b),Tuple{Array{Float64,1},Array{Float64,2}}}, ::Composite{NamedTuple{(:a, :b),Tuple{Array{Float64,1},Array{Float64,2}}},Tuple{Array{Float64,1},Array{Float64,2}}}) at /scratch/will/ml/IWVIHilbertGP/dev/VIEstimators/dev/ChainRulesCore/src/differential_arithmetic.jl:109
 [2] top-level scope at REPL[31]:1
caused by [exception 1]
MethodError: no method matching elementwise_add(::NamedTuple{(:a, :b),Tuple{Array{Float64,1},Array{Float64,2}}}, ::Tuple{Array{Float64,1},Array{Float64,2}})
Closest candidates are:
  elementwise_add(::Tuple, ::Tuple) at /scratch/will/ml/IWVIHilbertGP/dev/VIEstimators/dev/ChainRulesCore/src/composite_core.jl:86
  elementwise_add(::NamedTuple{an,T} where T<:Tuple, ::NamedTuple{bn,T} where T<:Tuple) where {an, bn} at /scratch/will/ml/IWVIHilbertGP/dev/VIEstimators/dev/ChainRulesCore/src/composite_core.jl:93
Stacktrace:
 [1] +(::NamedTuple{(:a, :b),Tuple{Array{Float64,1},Array{Float64,2}}}, ::Composite{NamedTuple{(:a, :b),Tuple{Array{Float64,1},Array{Float64,2}}},Tuple{Array{Float64,1},Array{Float64,2}}}) at /scratch/will/ml/IWVIHilbertGP/dev/VIEstimators/dev/ChainRulesCore/src/differential_arithmetic.jl:107
 [2] top-level scope at REPL[31]:1

Am I using the interface incorrectly, or is this a bug? I can't see any NamedTuple tests, so perhaps it's a bug?

[oxinabox]

Yeah, I never tested it with NamedTuple as primal so could well be a bug.
Will add tests then investigate