rand_tangent(adjoint(array)) returns Composite even though Adjoint is a natural differential

[mzgubic]

Presently

julia> A = rand(3, 3);

julia> adj = adjoint(A);

julia> ChainRulesTestUtils.rand_tangent(adj)
Composite{Adjoint{Float64,Array{Float64,2}}}(parent = [-0.5319309079526605 -1.2397774126992458 1.450107932767501; 0.2443697244819896 -0.5283784206385908 -0.36484818100635125; -0.18829408436246423 -1.0652797084400936 -1.0457119567770545],)

whereas we would like to get an Adjoint out, which is a natural differential type.

Probably just need to define a new dispatch for rand_tangent?

[oxinabox]

if we did JuliaDiff/ChainRulesTestUtils.jl#123 it would cover this implictly.

[willtebbutt]

Probably just need to define a new dispatch for rand_tangent?

Yes, this makes sense. Alternatively, we could just add a kwarg that says whether to produce a natural differential or not.

[oxinabox]

Alternatively, we could just add a kwarg that says whether to produce a natural differential or not.

Can't really do that because rand_tangent is mostly used implictity when the user doesn't provide a differential.
We need to chose one.
And a natural differential is the right choice if it is available.

[willtebbutt]

Can't really do that because rand_tangent is mostly used implictity when the user doesn't provide a differential.

Fair point.

And a natural differential is the right choice if it is available.

Is it? What about functions that you can AD automatically that involve Adjoint{Float64, Matrix{Float64}}? They'll Composites. I'm not claiming that the natural differential is the wrong choice, but it's definitely not the only choice.

[oxinabox]

I agree it isn't the only choice.
But I do claim it is the right one.
the reason we call it natural is it is the one humans use when writing rules.
This package is (primarily) for testing rules humans write.
and it works best if you don't mix natural and structural deriviatives
(we do need to solve that, but its not a problem for this package)

[willtebbutt]

This package is (primarily) for testing rules humans write.

This might be true right now, but it's not where we're headed. Quite near the top of the TODO list is making test_rrule and test_frule accept an AD as an argument, so that we can use them to test arbitrary bits of code.

the reason we call it natural is it is the one humans use when writing rules.

I agree that humans use this package to test rules, and humans seem to like making the tangents of arrays other arrays, but they write rules that are to be used inside AD.

AD much prefers Composites to natural differentials -- if a human doesn't test their rule using the thing that AD likes, then they've not tested their rule properly.

Moreover, part of the point of rand_tangent is to make it so that humans don't have to think hard about exactly what type the AD would like, they just make have rand_tangent generate it for them, and then they figure out how to deal with it in their rule.

[oxinabox]

I stand by the statement that we should do this.

AD much prefers Composites to natural differentials -- if a human doesn't test their rule using the thing that AD likes, then they've not tested their rule properly.

Possibly we should have an extra flag to also test the Composite.
but for array rules, its often super useful to accept array deriviatives.
That means you can use your own weird linear operations defined in your package on them.
I mean even things like mul! and muladd are optimized in LinearAlgebra stdlib for stuff like adjoint.
Forcing that stuff to be implemented for TangentAdjoint is kinda no-starter.
(At least until we can convert between structural and natural representations)

People want to write rules that way, we should let them.
The code is clearer when written using natural differentials.
We should sort it out so they always can write in the form they want.

And its perfectly common for them to only get array inputs, because the source of the cotangent is another rule.

Not having this means i have to write a custom tangent for like 20 tests.

[willtebbutt]

Fair enough. My takeaway from this is just that we

1. need to get projections sorted, and
2. need to use that functionality to make it straightforward to generate tangents in pretty arbitrary forms, because clearly we need both strutural and natural tangents.