Add sparse Hessian decompression

[ElOceanografo]

Following up on #142, this is a basic implementation of sparse Hessian decompression using matrix coloring. (See also SciML/Optimization.jl#269.) Basic interface is modeled on the existing one for Jacobians; it includes forwarddiff_color_hessian and forwarddiff_color_hessian! methods for in-place and out-of-place computations, and a ForwardColorHesCache type containing buffers for computation. Feedback welcome!

To do:

• Make sure there are method signatures corresponding to all the ones for Jacobians
• More tests
• Make sure performance is ok

[codecov-commenter]

Codecov Report

Merging #190 (e1c8eaf) into master (74d9fc6) will increase coverage by 1.20%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master     #190      +/-   ##
==========================================
+ Coverage   79.94%   81.14%   +1.20%     
==========================================
  Files          14       15       +1     
  Lines         753      801      +48     
==========================================
+ Hits          602      650      +48     
  Misses        151      151              

Impacted Files	Coverage Δ
src/SparseDiffTools.jl	100.00% <ø> (ø)
src/differentiation/compute_hessian_ad.jl	100.00% <100.00%> (ø)

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 74d9fc6...e1c8eaf. Read the comment docs.

[ElOceanografo]

forwarddiff_color_jacobian! has default arguments for colorvec and sparsity, which I currently don't supply for forwarddiff_color_hessian!. Currently not sure what makes the most sense:

1. defaulting to an assumption that the Hessian is dense (i.e., colorvec = eachindex(x), sparsity = ones(length(x), length(x))),
2. doing automatic sparsity detection when they aren't specified, or
3. leaving it as-is and making the user explicitly supply the coloring and sparsity pattern.

[ChrisRackauckas]

1

[ElOceanografo]

I currently have two signatures for forwarddiff_color_hessian!:

• forwarddiff_color_hessian!(H, f, x, colorvec, sparsity) (creates a ForwardColorHesCache)
• forwarddiff_color_hessian!(H, f, x, hes_cache) (totally non-allocating)

There are also two corresponding signatures for forward_color_hessian, which create the sparse Hessian H instead of modifying it in place. I think this matches the interface for forwarddiff_color_jacobian!. Are any others needed?

[ElOceanografo]

Ok, I think this is getting pretty close. @ChrisRackauckas when you've got a second, lemme know what else you think it needs and I'll try to take care of it!

[ChrisRackauckas]

Suggested change

	hes_cache.grad!(hes_cache.dG, x, hes_cache.grad_config)
	x .-= ϵ .* @view hes_cache.D[:, j]
	hes_cache.buffer[:, j] .= (hes_cache.dG .- hes_cache.G) ./ ϵ
	hes_cache.grad!(hes_cache.dG, x, hes_cache.grad_config)
	x .-= 2ϵ .* @view hes_cache.D[:, j]
	hes_cache.buffer[:, j] .= (hes_cache.dG .- hes_cache.G) ./ 2ϵ

If you're not going to use a cached f here, is there a reason to not do a central difference?

[ElOceanografo]

The only reason not to is because it modifies x three times as opposed to two, saving a little time. However, profiling shows that the time saved is only ~2% of the total, and the centered difference decreases error by more than 10x, so a worthwhile tradeoff. Changed as suggested.

Can you clarify what you mean by "use a cached f?"

[ChrisRackauckas]

Is this still WIP? Looks close to done.