A question about the negative buffer

[avis-ma]

Hi, I have a question about the buffer in class Adversary_Negatives. When initializing, the class registered two buffers, W and v. I know the W means the memory bank like MoCo, but what buffer v means. I can't understand very clearly. Kindly hope you can help me understand. Thank you very much.

[wang3702]

Please check https://pytorch.org/docs/stable/_modules/torch/optim/sgd.html#SGD. The v here is a mimic of the SGD momentum update to keep record of previous gradients to make the gradient update to be consistent.

v_{t+1} & = \mu * v_{t} + \text{lr} * g_{t+1},
p_{t+1} & = p_{t} - v_{t+1}.

we also further consider weight decay here.

[avis-ma]

Wow, I see. Thank you very much for your kindly reply.