Quasi-Newton update rules issues

[mateuszbaran]

I'm currently working on some updates to quasi-Newton direction update rules. Here are a few things to check/consider:

• QuasiNewtonCautiousDirectionUpdate doesn't seem to use θ function anywhere.
• QuasiNewtonLimitedMemoryDirectionUpdate doesn't seem to use scale.
• Some direction updates need a reset mechanism called by initialize_solver!, and maybe also when the direction stops being a descent direction. That shouldn't happen for a well-designed set of options but it's a bit hard to tell which ones are safe from that issue.
• Unify projections to a custom function instead of a boolean flag.

I'm also working on some variant of direction update for manifolds with corners inspired by L-BFGS-B. There are different approaches with elaborate line searches and update matrices but I'm aiming for something relatively simple and generic. I'm not aiming at competing with Fortran codes, my goal is making combined manifold-box constrained optimization work relatively well.

[kellertuer]

I started looking a bit into these points.

Ad 1) The function is used when updating the Hessian at

Manopt.jl/src/solvers/quasi_Newton.jl

Line 677 in 1856a26

bound = d.θ(norm(M, p, X))

or

Manopt.jl/src/solvers/quasi_Newton.jl

Line 738 in 1856a26

bound = d.θ(norm(M, p_old, get_gradient(mp, p_old)))

Ad 2) That might indeed be the case, I could only find the scale in the update of the full-matrix variants.

Manopt.jl/src/solvers/quasi_Newton.jl

Lines 514 to 516 in 1856a26

	if iter == 1 && d.scale == true
	d.matrix = skyk_c / inner(M, p, st.sk, st.sk) * d.matrix
	end

I am not sure where the scaling was “lost” in one of the reworks, but we can surely bring that back

Ad 3) Sure, I am not 100% sure how to realise that, but when we have the iterate available doing a reset when that is zero sounds reasonable. When we wrote this code, reusing the state (and hence the updates) was not so much thought given to, for sure.

• I also noticed that one of the recent updates that meant to introduce a stability idea did

Manopt.jl/src/plans/quasi_newton_plan.jl

Line 500 in 1856a26

project::Bool=true,

and

Manopt.jl/src/plans/quasi_newton_plan.jl

Line 582 in 1856a26

d.project && embed_project!(M, r, p, r)

which we basically have used before but then a bit more generic (not just a boolean)

Manopt.jl/src/solvers/truncated_conjugate_gradient_descent.jl

Line 18 in 1856a26

* `project!`: (`copyto!`) for numerical stability it is possible to project onto

and then

Manopt.jl/src/solvers/truncated_conjugate_gradient_descent.jl

Line 741 in 1856a26

tcgs.project!(M, tcgs.δ, p, tcgs.δ)

which might be a bit nicer to allow also for other ways to care for stability.
We should both unify this and probably also switch the second one to embed_project! (since that default is older than the embed/project discussion in Manifolds.jl).

[mateuszbaran]

Ad 1) The function is used when updating the Hessian at

I see, thanks.

Ad 3) Sure, I am not 100% sure how to realise that, but when we have the iterate available doing a reset when that is zero sounds reasonable. When we wrote this code, reusing the state (and hence the updates) was not so much thought given to, for sure.

I have it sketched already so I will only ask for a review when it's ready 🙂

We should both unify this and probably also switch the second one to embed_project! (since that default is older than the embed/project discussion in Manifolds.jl).

Yes, unifying that is a good idea.

[kellertuer]

If you see how to correctly add the scaling ( and unifying that to be a real value), feel free to add that as well. I am sure it was from one of the Wuang papers about QN.