Support LBFGSB.jl

[goerz]

Would it be possible to add support for LBFGSB.jl? It seems that currently, the only LBFGS implementation in Optimization.jl that supports box constraints is NLopt.LD_LBFGS(). There's also Optim.LBFGS, but that doesn't allow for constraints.

I'm finding NLopt.LD_LBFGS() behaving a little wonky (and I'm not sure how exactly it actually implements the box constraints), so it would be great to have LBFGSB (which explicitly supports bounds, hence the B suffix) for a second opinion.

Even without support for constraints: I've always found LBFGSB to be a great go-to optimizer because it has a very solid hard-coded linesearch, which is just one less thing to worry about: With, e.g., Optim.LBFGS, I've always had issues tweaking linesearch parameter to get similarly good convergence as LBFGSB gives me out of the box.

So I definitely wouldn't mind being able to call LBFGSB with Optimization's much more convenient high-level interface.

[cefitzg]

I agree with this feature request, and I think this is in development here: https://github.com/SciML/Optimization.jl/tree/lbfgsb

[cefitzg]

@Vaibhavdixit02, could you update the documentation or provide an example once the feature is implemented?

[Vaibhavdixit02]

yup that's the plan. I am delayed because I wanted to finish it up with an augmented lagrangian so it cna be used with non-linear constraints this is also quite close to done #727 but I need to make more type stable etc etc

[cefitzg]

Awesome, thanks!

[cefitzg]

@goerz, For what it's worth, I tried the implementation here: https://github.com/SciML/Optimization.jl/tree/lbfgsb. It seems to run much faster than Optim.LBFGS or NLopt.LD_LBFGS() for my problem.

[cefitzg]

@Vaibhavdixit02, is there a way I can get dev code before the stable release? In particular, I want to use LBFGSB in the context of this call for some ODE parameter estimation.

sol = solve(optprob, Optimization.LBFGS(), maxiters = 1000)

I installed the lbfgsb branch before it got merged (Pkg.add(url=https://github.com/SciML/Optimization.jl.git ,rev="lbfgsb")), but wanted to see if newer code has non-default ret values (success, failure, etc.). On lbfgsb branch sol.retcode = Default.

[Vaibhavdixit02]

I don't think I had changed that. Some compat issues are holding back a release with it right now.

[cefitzg]

@Vaibhavdixit02, I wanted to check in on timeline for this. I know there is a lot of uncertainty about this sort of thing, but I was wondering if you could give a ballpark, such as weeks timescale or months timescale? I'm hoping to use this implementation for a large scale computation.

[Vaibhavdixit02]

It should be available in the latest release

[cefitzg]

@Vaibhavdixit02, when I look at: sol.retcode I get Default status.

https://docs.sciml.ai/Optimization/dev/API/optimization_solution/

https://docs.sciml.ai/SciMLBase/stable/interfaces/Solutions/#retcodes

Is this being planned or would it be helpful to open an issue for this? I want to look at retcode to check optimization results.

[Vaibhavdixit02]

It's planned, I am not sure if the julia wrapper for LBFGSB returns the return code right now

[cefitzg]

@Vaibhavdixit02, do you know timescale (weeks, months), etc. for that retcode change. Open an issue or wait? Thanks for your help!