How to use it in huggingface AdamW optimizer?

[ztl-35]

hi, thanks for this work! I want use it in huggingface AdamW optimizer to train Pre-trained language model, such as BERT. How can I use it, thanks!

[TimDettmers]

You should be able to just replace AdamW with bnb.optim.Adam (which uses the AdamW formulation if your weight decay is > 0.0). Can you try that and report back if there are any issues?

[ztl-35]

Hi, I try this method, replacing the AdamW with bnb.optim.Adam. Although output of he task performance is normal, I find the result is much lower than the previous Huggingface AdamW function and I don't change any hyper-paramerters.

[TimDettmers]

Can you share the model that you are fine-tuning along with the dataset and results? If you are fine-tuning on GLUE for example there can be quite some noise in the results and the best comparison is done across 5-10 random seeds to get a clear picture of variability between 32-bit and 8-bit AdamW.

[TimDettmers]

The problem in this example was that the default weight_decay argument is different from AdamW and Adam8bit. Even though Adam8bit implements the same algorithm by default weight decay is turned off. The new AdamW8bit class should rectify this since it has the correct default hyperparameter. This was fixed in 2f8083b.