New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
EMA update on CosineCodebook #26
Comments
Would you like to explain why ema does not work for the unit-normalized codebook? |
I found when using EMA for cosine code book, the l2-norm of the input to the vq module would grow gradually, from 22 -> 20000, leading to growing training loss. Has anyone met this problem? |
In case anyone else has this problem, I add a |
@Saltychtao I also encounter a similar issue. Does vq_in refer to VectorQuantize.project_in? |
Yes. |
@Saltychtao Hi, just want to make sure that the current vesion of implementation here seems to put one normalization (l2norm) after the project_in. I also encounter the training loss explosion issue somehow at current version |
@santisy want to try turning this on (following @Saltychtao 's solution) let me know if it helps |
The original VIT-VQGAN paper does not seem to use EMA update for codebook learning since their codebook is unit-normalized vectors.
Particularly, to my understanding, EMA update does not quite make sense when the encoder outputs and codebook vectors are unit-normalized ones.
What's your take on this? Should we NOT use EMA update with CosineCodebook?
The text was updated successfully, but these errors were encountered: