Clarify documentation on sparse second-order differentiation

[ElOceanografo]

Currently the documentation is not totally clear whether you should do AutoSparse(SecondOrder(OuterBackend(), InnerBackend(), ...)) or SecondOrder(AutoSparse(OuterBackend(), ...), InnerBackend()). Based on some quick timing experiments, I assume it's the former...would be good to have it laid out explicitly.

It would also be good to have a brief description of the difference between AutoSparse(InnerBackend(), ...) and AutoSparse(OuterBackend(), InnerBackend(), ...), i.e. when you'd want to do one or the other and the advantages/disadvantages.

[gdalle]

Currently the documentation is not totally clear whether you should do AutoSparse(SecondOrder(OuterBackend(), InnerBackend(), ...)) or SecondOrder(AutoSparse(OuterBackend(), ...), InnerBackend()).

Indeed you should wrap the whole backend inside AutoSparse, like so:

AutoSparse(SecondOrder(Outer(), Inner()))

I'll add it to the SecondOrder docstring.

It would also be good to have a brief description of the difference between AutoSparse(InnerBackend(), ...) and AutoSparse(OuterBackend(), InnerBackend(), ...),

I'm not sure what you mean here, can you perhaps fix the typo or forgotten SecondOrder?

[ElOceanografo]

What I mean is, you can do e.g. AutoSparse(AutoZygote()) or AutoSparse(SecondOrder(AutoForwardDiff(), AutoZygote())). Both will work and produce a sparse Jacobian/Hessian, but it's not clear what exactly is the difference algorithmically. Is the first one basically equivalent to AutoSparse(SecondOrder(AutoZygote(), AutoZygote()))?

[gdalle]

Yes, this behavior is independent from AutoSparse. When a second order operator is not implemented by a given backend, then second_order_operator(f, backend, x) falls back on second_order_operator(f, SecondOrder(backend, backend), x). But you lose the benefits of preparation in this case, so it's better to specify a SecondOrder backend right away if you are sure that the backend alone will not suffice. I'll add this to the docs too

[gdalle]

@ElOceanografo I made an effort to clarify the documentation on sparse, second order and their interactions. This is visible in the dev docs, and in the docs of the newly-released DifferentiationInterface v0.4.2. Does it clarify things for you? Are there still uncertain aspects?

[ElOceanografo]

Yes, that's much clearer, thanks!