Skip to content
This repository has been archived by the owner on Nov 17, 2023. It is now read-only.

Optimizer wish list #9182

Open
1 of 2 tasks
sxjscience opened this issue Dec 22, 2017 · 8 comments
Open
1 of 2 tasks

Optimizer wish list #9182

sxjscience opened this issue Dec 22, 2017 · 8 comments

Comments

@sxjscience
Copy link
Member

sxjscience commented Dec 22, 2017

Optimizer plays a fundamental role in machine learning. This issue tracks the optimizers that people have requested for support in MXNet. We can also comment below to recommend new optimizers. It's inspired by the thread in https://discuss.gluon.ai/t/topic/3714.

@szha @szhengac

@VikingMew
Copy link

NeumannOptimizer

@kobenaxie
Copy link

AdamW, SGDW

@ZiyueHuang
Copy link
Member

working on FTML

@jasonyu1996
Copy link
Contributor

  • Limited-memory BFGS (L-BFGS)

@sxjscience
Copy link
Member Author

sxjscience commented Jun 28, 2019

@braindotai
Copy link

braindotai commented Jul 16, 2019

AdaBound
The authors call it as fast as Adam and as good as SGD.
So it might a good one.

@Hunter-Zolomon
Copy link

@szha I would like to attempt an implementation of the RAdam optimizer. I have written some code here that probably isn't yet be ready for a PR (might not even be correct), so how would you like me to share it with you for a review so that you can point me in the right direction? Thanks.

@szha
Copy link
Member

szha commented Nov 19, 2021

@Hunter-Zolomon thanks for offering to contribute.

To implement optimizer in MXNet, the best reference would be the existing optimizers. You can find many in python/mxnet/optimizer folder.

Once you have the implementation ready, you can refer to the contribution guides and submit a pull request. Make sure to also include tests for the new optimizer following examples here. Feel free to ping me or others for review on the pull request.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

8 participants