NLS: obj, grad! and objgrad! without residual evaluation

[tmigot]

There exist functions re-using the residual memory

f = obj(nls, x, Fx)
g = grad!(nls, x, g, Fx)
f, g = objgrad!(nls, x, g, Fx)

However, all of them internally call residual!. It would be great to also have similar functions that don't evaluate the residual.

In a solver context, often we will call residual! and then want to evaluate the objective, gradient, etc.
Any opinion on this @dpo @abelsiqueira @amontoison @geoffroyleconte @paraynaud ?

[abelsiqueira]

Makes sense. It could be just a keyword argument, what do you think?

[amontoison]

I like the proposition of Abel, we could add a keyword argument to reuse the value of the Fx vector without calling residual!. If x was not updated, we don't need to call residual! multiple times.

[paraynaud]

A keyword should be enough, but do you prefer

f = obj(nls, x, Fx; recompute_Fx::Bool=true)

or

f = obj(nls, x; Fx=residual(nls))

while keeping the current f = obj(nls, x, Fx).
I am not sure what is best but neither of the two solutions works without making some changes in the Solvers.

[amontoison]

The first version with the keyword argument will not break the API. I prefer it.

[tmigot]

Thanks for the comments. I like @paraynaud first suggestion too. I was wondering about this

obj(nls, x, Fx, shah::UInt64)

with the idea

function obj(nls, x, Fx, shah::UInt64)
  if hash(x) != shah
    return obj(nls, x, Fx) # recompute Fx
  else
    return dot(Fx, Fx) / 2
  end
end

[abelsiqueira]

I also prefer to not break the current API, so the first suggestion.

Maybe the hash idea could be a CacheNLPModel? Because it makes sense in a few situations (there was talk about it in the past, with the name "memoization"). Just for this case it seems out of place. It would be better to be explicit with a keyword argument like recompute_Fx or skip_residual_call. Also, it might be the case that specific model implementations have a different obj than just the dot(Fx, Fx) / 2, like adding a regularization term, and then it becomes complicated to make sure that you are dispatching correctly.

[dpo]

Just recompute=true would do it.

The hash idea should be benchmarked carefully. I tried recently and found it slower than comparing vector components one by one even for fairly large n.