Reading: All-but-the-Top: Simple and Effective Postprocessing for Word Representations

[a1da4]

0. Paper

@inproceedings{
mu2018allbutthetop,
title={All-but-the-Top: Simple and Effective Postprocessing for Word Representations},
author={Jiaqi Mu and Pramod Viswanath},
booktitle={International Conference on Learning Representations},
year={2018},
url={https://openreview.net/forum?id=HkuGJ3kCb},
}
[paper]

My code [GitHub]

1. What is it?

They proposed an effective post-process method for word embeddings.

2. What is amazing compared to previous studies?

They claimed that there are many embedding methods and their performance are approximately same.
Moreover, the authors points 2 problems in PMI-based methods(word2vec, sppmi-svd).

• not of zero-mean: any word w, its vector v(w) is not in center.
• not isotopic: there are some bias in vector distribution.

3. Where is the key to technologies and techniques?

D is a parameter, approximately d/100. d is a total dimension.

4. How did validate it?

They defined the way to calculate isotopy, I({v(w)}).

They used Word2vec and Glove, their method improved the isotopy.

Moreover, their methods improved word embedding tasks.

5. Is there a discussion?

6. Which paper should read next?