Plugging vector-quantize-pytorch into taming-transformers

[tanouch]

Hi,

I noticed your architecture could be plugged within the pipeline from https://github.com/CompVis/taming-transformers. I have proposed a code here (https://github.com/tanouch/taming-transformers) doing that. It enables to properly compare the different features proposed in your repo (Lower codebook dimension, Cosine similarity, Orthogonal regularization loss, etc) with the original formulation.

The code from this repo can be seen in both files

• taming-transformers/taming/models/vqgan.py
• taming-transformers/taming/modules/vqvae/quantize.py

As you can see, it is easy to launch a large scale training with your proposed architecture.

I am not sure this issue belongs here or in the taming-transformers repo. However, I thought you might be interested.
Thanks again for your work and these open-sourced repositeries !

[lucidrains]

@tanouch thank you! i'm also open to redesigning the base class (or adding an adapter wrapper on top) that would allow it to plug into the taming transformers library more easily, if you think that makes sense

[tanouch]

Hi @lucidrains,
Thanks for your response... So the way I understand it is that your current base/main class "class VectorQuantize" is the equivalent of the class "class VectorQuantizer" from (https://github.com/CompVis/taming-transformers/blob/master/taming/modules/vqvae/quantize.py).

This consequently makes the two following classes the same...(and redundant)

• class VQModel (from TT)
• class VQLucidrainsModel (deriving from your quantizer)
  (here: https://github.com/tanouch/taming-transformers/blob/master/taming/models/vqgan.py)

Maybe, we could simply add an argument in the constructor of the VQGAN class specifying what quantizer are we going to use...
What do you think ?