Depereciation warning in pytorch 1.5 (maybe and above?)

[Junyoungpark]

Hi! I encountered following warning while I using AdamP for my project

..\torch\csrc\utils\python_arg_parser.cpp:756: UserWarning: This overload of add_ is deprecated:
	add_(Number alpha, Tensor other)
Consider using one of the following signatures instead:
	add_(Tensor other, *, Number alpha)

Might this be relevant to the AdmaP update?

[bhheo]

Hi

We implement AdamP based on PyTorch 1.3
PyTorch 1.5 changes default args of in-place operations such as add_() and addcmul_().
So, following lines of AdamP makes the warnings

AdamP/adamp/adamp.py

Lines 80 to 81 in 64a6310

	exp_avg.mul_(beta1).add_(1 - beta1, grad)
	exp_avg_sq.mul_(beta2).addcmul_(1 - beta2, grad, grad)

AdamP/adamp/adamp.py

Line 101 in 64a6310

p.data.add_(-step_size, perturb)

If you change it as

 exp_avg.mul_(beta1).add_(grad, alpha=1 - beta1) 
 exp_avg_sq.mul_(beta2).addcmul_(grad, grad, value=1 - beta2) 

 p.data.add_(perturb, alpha=-step_size) 

then the warnings will disappear
But, I'm not sure whether this changes is compatible with old versions. So, I will not apply the changes to master or pip.

You can also find difference in official AdamW.py
Torch 1.4
https://github.com/pytorch/pytorch/blob/74044638f755cd8667bedc73da4dbda4aa64c948/torch/optim/adamw.py#L100-L101
Torch 1.5
https://github.com/pytorch/pytorch/blob/4ff3872a2099993bf7e8c588f7182f3df777205b/torch/optim/adamw.py#L104-L105

[Junyoungpark]

thanks for the comments! I'll use the code snippet for torch 1.5

[bhheo]

Inplace operation was changed in #7.
Thanks for your contribution.

[SanghyukChun]

@Junyoungpark
We just release adamp==0.3.0(https://github.com/clovaai/AdamP/releases/tag/v0.3.0).
Please retry pip install adamp --upgrade :)