You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
@JaheimLee Hi, I did not test with fine-tune bert, bert is too large for me. I tested a small transformer here https://github.com/juntang-zhuang/fairseq-adabelief , it seems the default in adabelief-pytorch==0.2.0 works. eps=1e-16 helps. I'm not so sure about rectify, sometimes it helps sometimes not, perhaps need some tuning. Other hyper-parameters, such as lr, beta, the same value as Adam works.
BTW, if you use fp16 to accelerate, v0.2.0 might be problematic because eps=1e-16 is rounded to 0 in fp16. A by-pass is to forward and backward in fp16, but update parameter in fp32. See the link below #31 (comment), we are considering adding this to the next release
Have you ever tested adabelief for fine-tuning bert models? And what's the recommended hyper-parameters?
The text was updated successfully, but these errors were encountered: