Issue with OptimizationManopt initial Hessian #1036
Description
The latest OptimizationManopt bump left a TODO in the tests saying:
#TODO: This autodiff currently provides a Hessian that seem to not procide a Hessian
# TR Fails but also AD before that warns. So it passes _some_ hessian but a wrong one, even in format
@kellertuer can you elaborate a bit? My guess is that the initialization creates the problem and solver with a Hessian matching type and sparsity pattern in order to generate the types used in an iterator of some sort, then it procedures to update values, run a step, update values, run a step, etc. So if I had to guess what this is referring to, I would assume there is some construction where H is all zeros (or undefined values) because it's similar(prob.f.hess_prototype) directly as part of a type building phase, but Manopt then runs a first step in that phase rather than separating out the type construction from the initialization of the first step.
While that would be my general guess here given other common issues etc., this doesn't seem to actually work out because
Manopt.jl doesn't use an iterator interface, it uses a solve interface, so it shouldn't have this kind of initialization phase vs first solve kind of split anyways, it would always have to init and thus blank values in the Hessian shouldn't ever be an issue. This makes me a bit confused.
@kellertuer is it documented that in Manopt hessF is supposed to have the values of the Hessian at the initial condition point or something of the sort?