Cannot Converge with L2 Loss

[kingnobro]

I am trying to quantize the latent vector. To be specific, I use a Encoder to get the latent representation z of the input. Then I try to quantize z, then send z into Decoder.

However, during my experiment, I found the reconstruction loss cannot decrease with L2 loss, namely, the EuclideanCodebook. The model can converge with cosine similarity. Have any idea about this phenomenon?

I think cosine similarity only considers the direction of the vector, instead of the scale of the vector. I still want to use EuclideanCodebook.

[lucidrains]

@kingnobro are you on the latest version of the library?

[kingnobro]

@kingnobro are you on the latest version of the library?

Oh, I found the code was downloaded about three months ago. After using the latest version, the problem is solved.

[lucidrains]

@kingnobro ok great!

[IISCAditayTripathi]

@lucidrains @kingnobro, what was the problem? What changes in the code in the newer version made it work?