least_squares cost tolerance

[sapphire008]

Is there a way to specify a tolerance on the cost rather than ftol/gtol/xtol?

When using least_square to find multivariate, bounded roots (please suggest alternatives if it exists in scipy.optimize), I find that sometimes, it may be easier to terminate when the cost function is small (say 1E-3) than when ftol / gtol / xtol is small.

[nmayorov]

@sapphire008

Hi! No there isn't currently.

I understand your issue, but as it is not standard thing for least squares minimization (see http://ceres-solver.org/nnls_solving.html#solver-options for example) it wasn't implemented.

How would you go about implementing such criterion?

[sapphire008]

Here is the reason why I think that a tolerance on the cost would be nice.least_squares seems to offer a really robust way to solve for multivariate roots of nonlinear system of equations with bounds. When using it to find roots, we would expect that the cost function (which is always positive in this case) would approach zero. This is not the same as ftol, where it is the change of cost function. I have encounter many cases in my specific applications where ftol is reached (say 1E-12), but the cost function does not reach a value close enough to zero. It would require some additional iterations to bring the cost function close to zero. On the other hand, I have also encountered cases where ftol/gtol / xtol are not reached, but the cost function is a value that is enough to zero (say, cost < 1). In this case, specify a loose tolerance on the cost would help with early termination. To make least_squares more friendly for finding roots of nonlinear system of equations, I think a parameter like ctol would be great to have.

Since least_squares right now is purely in Python, I would think adding a ctol parameter would be feasible. For instance, in the trf.py file, in the function trf_bounds, around line 390, around the if termination_status is None, we can add something like

if cost < ctol:
   .... # termination clean up
    break

This could terminate the while loop and stop the iterations when cost is within a cost tolerance.

[nmayorov]

Need to decide on one thing - how to implement interaction between ftol, xtol, gtol and ctol. I think the most reasonable is to separate them completely, i.e. when ctol is set other tolerances are ignored. Do you agree?

Also 'lm' won't support ctol.

Do you want to implement this feature? I will help reviewing and maybe give some tips if necessary.

[sapphire008]

I think this is a great idea to separate ctol completely. I will try to make a pull request once I get it done. I will definitely need your help for tips. Thanks!

[mdhaber]

Oops, mis-click.
Many custom stopping criteria can be requested, so I'd suggest a more general solution: add a callback interface to least_squares like we already have in most minimization routines. (Actually, I'd suggest following the interface that is being retrofitted into other optimizers; e.g. gh-17486.) To implement ctol, the user can get the current value of the objective function from the OptimizeResult passed into the callback and raise StopIteration to terminate optimization.