Addition of FAIR's MADGRAD Optimizer

[DarshanDeshpande]

Describe the feature and the current behavior/state.

Relevant information

• Are you willing to contribute it (yes/no): yes
  If you wish to contribute, then read the requirements for new contributions in CONTRIBUTING.md
• Are you willing to maintain it going forward? (yes/no): yes
• Is there a relevant academic paper? (if so, where): yes. Aaron Defazio and Samy Jelassi, 2021
• Does the relavent academic paper exceed 50 citations? (yes/no): no
• Is there already an implementation in another framework? (if so, where): Pytorch, JAX
• Was it part of tf.contrib? (if so, where): no

Which API type would this fall under (layer, metric, optimizer, etc.)
Optimizer

Who will benefit with this feature?
The Madgrad algorithm, presented by Facebook's AI Research Lab, consistently matches or outperforms Adam and SGD algorithms. The code for this algorithm is available for Pytorch and JAX. The addition of this optimizer to TF-Addons will make this implementation available for Tensorflow users.

[tomerk]

Ecosystem review check: This is the sort of recent specific optimizer implementation that we don't usually add to core tensorflow/keras. If this is sufficiently widely used for addon's citations threshold then addons would be a better fit.

[DarshanDeshpande]

@tomerk I understand completely. I just thought that Tensorflow should have this implementation since JAX and PyTorch already do. If you want me to close this issue then please let me know :)

[seanpmorgan]

TensorFlow Addons is transitioning to a minimal maintenance and release mode. New features will not be added to this repository. For more information, please see our public messaging on this decision:
TensorFlow Addons Wind Down

Please consider sending feature requests / contributions to other repositories in the TF community with a similar charters to TFA:
Keras
Keras-CV
Keras-NLP