Addition of a new optimizer, LION

[old-school-kid]

If you open a GitHub issue, here is our policy:

It must be a bug, a feature request, or a significant problem with the documentation (for small docs fixes please send a PR instead).
The form below must be filled out.

Here's why we have that policy:.

Keras developers respond to issues. We want to focus on work that benefits the whole community, e.g., fixing bugs and adding features. Support only helps individuals. GitHub also notifies thousands of people when issues are filed. We want them to see you communicating an interesting problem, rather than being redirected to Stack Overflow.

System information.

TensorFlow version (you are using): tf-nightly
Are you willing to contribute it (Yes/No) : Yes

Describe the feature and the current behavior/state.
Google released a new optimizer, LION which is purportedly better than Adam and Adafactor, and is more memory-efficient too.

Will this change the current api? How?
Will a new optimizer essentially tf.keras.optimizers.Lion.

Who will benefit from this feature?
Everyone who uses Keras and is updated with recent developments in the field.

• Do you want to contribute a PR? (yes/no): Yes

[innat]

Src code is released https://github.com/google/automl/tree/master/lion

[nkovela1]

@xiangning-chen @crazydonkey200 wanted to see if your team would be interested in the implementation of this?

[crazydonkey200]

Sure. We will take a look. Since we have shared the TF2 and TF1 implementation in the open source repo, it is also good if other people would like to contribute the implementation and we can help review it.

[xiangning-chen]

Since Lion is already in the codebase, is it possible to also add it to the Keras API doc? Currently I don't see it here. @nkovela1 Thank you!

[nkovela1]

@xiangning-chen thanks for contributing the LION optimizer for Keras! The Keras API doc on keras.io will not have the LION optimizer added until the stable release of TF 2.13 that includes it, which is many months from now. This is because users don't have access to the said optimizer until then. Thanks again!