L-BFGS for HessianUpdateStrategy

[MarcYin]

When the input vector is large, BFGS and SR1 HessianUpdateStrategy give memory error. Is it possible to use L-BFGS for the Hessian update, which should reduce the memory requirements for Hessian.

[MarcYin]

Any updates?

[andyfaff]

First, I assume that you are using trust-constr, as HessianUpdateStrategy is only available for that method. Only SR1 and BFGS are available at this time. If you were to implement a class that inherits HessianUpdateStrategy and implements its methods then you would also be able to use that.
Is it imperative that you use trust-constr?

[MarcYin]

Hi, yes, I am using trust-constr and the existing Hessian update strategies can easily exceed the memory limit of normal computer when large scale optimisation required. Adding L-BFGS hessian update will make the trust-constr a more general optimisation algorithm. I tried to code it up, but have not been able to do it...I just wandering if anyone is familiar with the L-BFGS hessian update and could help on this.

[andyfaff]

@MarcYin, is there a reason why you can't use L-BFGS-B directly?

[MarcYin]

Hi, it seems that the trust region method is more robust when the optimised variables are highly correlated compared to the L-BFGS-B from here: http://fa.bianp.net/blog/2013/numerical-optimizers-for-logistic-regression/.

In my case, I have a large number of variables (more than 300,000) that are highly correlated, so I want to have a go with the trust region method but with limited memory requirement, to see if there is an improvement on the reduction of iterations or total optimisations time.

[paulestano]

Hi what is the status on this? If the documentation is correct, lots of the scipy.optimize optimizers do now support HessianUpdateStrategy. There is also now some evidence in the literature that L-BFGS Hessian update is robust enough to be used in other scenarios than the L-BFGS-B paper (there is for instance, this paper dealing with this). I think it could be useful to port it into HessianUpdateStrategy. I'd be happy to submit something if that's useful

[dschmitz89]

Hi again @paulestano , in general you can pick up any issue as long as there is no open PR yet. :)