Add adahessian

[bratao]

Hello,

I´m a big fan of this project. Recently a new optimized has been proposed, that promises SOTA for many tasks.

https://github.com/amirgholami/adahessian

The possibility of using it here is very good!

[jettify]

Thanks! I will take a look.

[jettify]

Since authors licensed code under unfriendly GPL V3, I can not reuse that code here and unfortunitly have to reproduce new implementation from paper.

Commercial usage of of authors implementation is limited, since forces users to license potentially commercial code under GPL also.

[bratao]

@jettify this is a re-implementation from scratch of the paper, with a MIT license
https://github.com/davda54/ada-hessian

[amirgholami]

Hi @jettify,

First, thanks so much for creating this repository, it is a great resource. I am one of the Adahessian authors, and we have just changed the license to the more friendly MIT License. I also just opened a new pull request that adds Adahessian optimizer to your repository:

#239 (comment)

I got the following results when I ran the optimizer on the Rosenbrock and Rastrigin test:

• Rosenbrock: best loss: 0.002011366948950946, best lr= 32.26606775899826
• Rastrigin: best loss: 2.3379755056396334e-05, best lr= 23.15826699019325 (best loss value among the current optimizers)
  P.S: Note that by design second order optimizers need larger learning rates

I have also attached the visualization for each test:

P.S: Based on the results I got, it seems Adahessian gets the best loss value among the optimizers

P.S2: @bratao Thanks so much for adding the Adahessian optimizer and for promoting it.

[jettify]

Just merged PR with optimizer, all thanks to @amirgholami !

[jettify]

PyPI is also updated https://pypi.org/project/torch-optimizer/0.1.0/