Why do you use cosine weight decay?

[CoinCheung]

Hi,

Thanks for providing the code. I found that you are also using cosine annealing curve to decay weight decay. It seems that the paper does not mention this. Would you please tell me why do you use it ?

[pengzhiliang]

Hello, follow BEiT, I implement the cosine weight decay strategy in this repo. However, I didn't use it.

You can check the "train_weight_decay" item in log