Skip to content

morganmcg1/AdahessianFastai

master
Switch branches/tags

Name already in use

A tag already exists with the provided branch name. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. Are you sure you want to create this branch?
Code

Latest commit

 

Git stats

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
 
 
 
 

Adahessian in Fastai

This is an implementation of the AdaHessian optimizer in fastai. Adahessian is the first, practically useful, second-order optimizer to have been developed and has shown impressive performance, especially within NLP where it consistently out-performs AdamW, the field's current default optimizer.

You can find the author's implementation along with a guides about second-order methods here

Author's citation:

@article{yao2020adahessian,
  title={ADAHESSIAN: An Adaptive Second Order Optimizer for Machine Learning},
  author={Yao, Zhewei and Gholami, Amir and Shen, Sheng and Keutzer, Kurt and Mahoney, Michael W},
  journal={arXiv preprint arXiv:2006.00719},
  year={2020}
}

About

No description, website, or topics provided.

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published