Workaround mxnet nd.topk regression

[leezu]

apache/mxnet#11271

Description

The behavior of nd.topk under the presence of nan values changed, causing wrong results in the pretrained word embeddings notebook due to def norm_vecs_by_row(x) inducing nan values for 0 word vectors. This PR adds a small epsilon to def norm_vecs_by_row(x) so that nan values are avoided.

Checklist

Essentials

• Changes are complete (i.e. I finished coding on this PR)
• All changes have test coverage
• Code is well-documented

Changes

• Fix embedding.ipynb for mxnet v1.3

[mli]

Job PR-153/1 is complete.
Docs are uploaded to http://gluon-nlp-staging.s3-accelerate.dualstack.amazonaws.com/PR-153/1/index.html

[astonzhang]

We need to keep k+5 to avoid retrieving special tokens.

[leezu]

The reason why special tokens were retrieved is that their word vectors were nan and the topk operator considered nan as the highest ranking. With the changes here the special tokens will not be among the topk elements, consequently instead of k + 5 we need k + 1 to only exclude the input token.

[leezu]

With mxnet 1.3 the topk operator however gets confused by the nan values and returns some 'random' words instead of the topk words if nan is present in the input.

[astonzhang]

ok. Do you know why nan is returned for special tokens? Is it because they are initialized as zero vecs so denominator of cosine sim has 0?

[leezu]

Yes, they are initialized as 0 and consequently there was a division by 0 in norm_vecs_by_row

[astonzhang]

same as above

[astonzhang]

Besides, please update get_top_k_by_analogy as

def get_top_k_by_analogy(token_embedding, k, word1, word2, word3):
    word_vecs = token_embedding.get_vecs_by_tokens([word1, word2, word3])
    word_diff = (word_vecs[1] - word_vecs[0] + word_vecs[2]).reshape((-1, 1))
    vocab_vecs = norm_vecs_by_row(token_embedding.idx_to_vec)
    dot_prod = nd.dot(vocab_vecs, word_diff)
    indices = nd.topk(dot_prod.reshape((len(token_embedding), )), k=k,
                      ret_typ='indices')
    indices = [int(i.asscalar()) for i in indices]
    return token_embedding.to_tokens(indices)

Otherwise, LGTM.

[leezu]

Started review by accident

[leezu]

Yes, they are initialized as 0 and consequently there was a division by 0 in norm_vecs_by_row

[leezu]

The get_top_k_by_analogy in your last comment doesn't work due to some missing functionality in TokenEmbedding. Please confirm that you are fine with ed0b9e5

[astonzhang]

LGTM

[mli]

Job PR-153/5 is complete.
Docs are uploaded to http://gluon-nlp-staging.s3-accelerate.dualstack.amazonaws.com/PR-153/5/index.html