Commitment Loss Problems

[pedrocg42]

Hello,

First of all, thank you so much for this powerful implementation.

I have been researching to train some VQ-VAE to generate faces from FFHQ 128x128 and I always have the same problem if I use the commitment loss (0.25) and the gamma (0.99) like in the original paper, the commitment loss seems to grow infinitely. I know you said that it is an auxiliary loss and that is not that important but is this normal behavior? If not, how can I avoid for that to happen in the case I wanted to use this loss?

Thank you so much in advance!

[kingnobro]

After reading the code, I notice the commitment loss is calculated using MSE, which measures the difference between quantize and x. It seems reasonable that MSE will grow larger, since the codebook is updated continuously. I think the point is, you have no reason to expect them to be similar. If the loss keeps decreasing, then the quantized result will have no difference with the original x.

Hope I explain my idea clearly.

[pedrocg42]

I totally agree with what you say but the purpose of the commitment loss is for the encoder embeddings to "commit" in other words, be similar to at least one of the codes of the codebook. I think ideally the encoder outputs should be similar (I don't know how similar) to the codes of the codebook. EMA (Exponential Moving Average) should "push" the codes to be similar to these embeddings and this commitment loss should "push" the embeddings to be similar to the codes until they converge. In my case, it seems like the commitment loss grows indefinitely and it does not converge. Is this right?

[kingnobro]

Oh your explain helps deepen my understanding about VQVAE. Currently I have the same problem, namely the infinite commitment loss. I tried to set the commit_weight to a small number but it failed. So I just remove it.

Hope someone know the answer :)

[danieltudosiu]

@pedrocg42 that happened to us as well, what is your decay and commitment_cost?

[lucidrains]

i've seen a number of papers omit commitment loss

i don't think it is necessary

[pedrocg42]

Right now I am following @lucidrains recommendation of not using any commitmen_loss and just using EMA to update the VQ codebooks. At the time I tried several configurations of both decay and commitment_cost with the same outcome, commitment loss growing continuously. The range I tried for decay was 0.5-0.99 being 0.99 my starting point, and the range for the commitment cost I tried was 0.5-2. The results only using EMA are good though, so I recommend you to not worry that much even if you use the commitment cost and your commitment loss is huge.

[IISCAditayTripathi]

It's great that it worked without using the commitment loss. However, why would the commitment loss increase continuously?