Vector Quantization Layer

[Simsso]

Development of a production-ready vector quantization (VQ) layer in TensorFlow, based on the prototype developed in #25 and merged with #52 (+prototype 2).

Development branch: vq-layer

Documentation

Sub-tasks

• Test batch size effect on gradient (f390595)
• Assignment of least used embedding space vectors with values from input x that were furthest away from embedding space vectors
• Consider developing a custom C++ op
• Research on current memory efficiency (which parts consume a lot of RAM, how to profile)
• is_training parameter updates only during training (not needed anymore)
• Add scatter_update call to tf.GraphKeys.UPDATE_OPS
• dotp norm order VQ-Layer Cosine Distance #63

[Simsso]

Just discovered the tf.test.TestCase class, which we should consider using.

[Simsso]

Since L1, L2, and Inf norm seem to be problematic in high dimensional space, it might be worth considering to add an additional norm order, namely dotp. It would

1. normalize all input vectors to unit norm
2. initialize the embedding space with unit norm vectors
3. replace inputs with the vector from the embedding space to which the dot product is the greatest
4. define the loss as the negative dot product

Issue for that #63

[Simsso]

I've noticed that the VQ layer uses some of the embedding vectors very sparsely. Despite beta-loss (beta=0.5, alpha=1). Here, gamma=0:

That's something worth contemplating about. Maybe the idea of replacing less frequently used embeddings should be pursued.

[FlorianPfisterer]

My guess is that L1/L2/inf norm are just not good distance measures in high-dimensional space. Maybe we could try applying PCA or something to both activations and embeddings and then calculate the distance on their basis.
-> will be discussed in #56