[Feature Request] Implementation of L-BFGS optimizer

[jasonyu1996]

Hi!
Would the developers consider adding an implementation of limited memory BFGS to the library? PyTorch offers one here: https://pytorch.org/docs/stable/optim.html?#torch.optim.LBFGS

[adaaaaaa]

follow with interest...

[lanking520]

@jasonyu1996 thanks for your question and keep watching this thread.
@eric-haibin-lin do we have something similar in MXNet?

@mxnet-label-bot please label this as [feature, memory]

[anirudhacharya]

MXNet does not have this optimizer( see here), and optimizer requests are tracked here - #9182

[jasonyu1996]

Thanks! I have added it to the optimizer request list. Actually I am quite willing to see if I could implement one for MXNet myself.

[haojin2]

@jasonyu1996 You're more than welcome to add this thing yourself if you want to, most of the optimizers should be here: https://github.com/apache/incubator-mxnet/blob/master/python/mxnet/optimizer.py. You can use any of them (for example, SGD) as reference to implement this. And all unit tests for optimizers are here: https://github.com/apache/incubator-mxnet/blob/master/tests/python/unittest/test_optimizer.py. Please make sure to add a test for your new optimizer once you finish implementing it.