Could you please implement a Adafactor optimizer? :)

[christophschuhmann]

🚀 Feature

Could you please implement a Adafactor optimizer? :)

( https://arxiv.org/abs/1804.04235 )

Motivation

In contrast to Adam it requires much less GPU memory.
I tried to use the FairSeq implementation for the pytorch-transformers, but I'm no expert and I couldn't get it done.

Could you please do that? :)

Additional context

[thomwolf]

What didn't work for you with the fairseq implementation?

It seems pretty self-contained: https://github.com/pytorch/fairseq/blob/master/fairseq/optim/adafactor.py#L65-L213

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[moscow25]

FYI @sshleifer -- I was wrong -- able to train T5-large even batch==1 with FP32, no gradient check-pointing and ADAM. Given that T5 team strongly recommends AdaFactor -- giving it a try, other pieces perhaps being more difficult...