Position Encoding

[anonymous012345678]

Hi domluna,
How did you get the equation in position_encoding? It seems different from the one in the paper, unless I made a silly algebra mistake...
Even then, is there an advantage in splitting out the equation into the way you wrote it? Some sort of optimization?

[domluna]

It came from https://github.com/facebook/MemNN/blob/master/MemN2N-babi-matlab/build_model.m#L9. It's been a while but I think it worked better than what was written in the paper. I'm not sure why they did it that way, didn't explain it afaik.

[domluna]

If anything it was a slight improvement, I don't think it made much of a difference.

[anonymous012345678]

Thank you.

[kodakfu]

When i implement the equation in paper and the way in code , it seems that there are some commen feature in the result.

• The result is matrix with 2 dimensions and its shape is [sentence_size, embedding_size] . when compare the matrix from 2 ways, the value decreases as the index of word increased until the medium index of matrix, and increase as the index of word increased until final index.

• It means that the value in matraix are symmetrical. when i first find the rule in the matrix, it arouses my curiosity.

• i read the data and it seems that the index of all the answer is first of the sentence or last of the sentence.Therefore, the formular is result from the data ,in other words, the postion encoding is another attention which makes the first or last word more important

• So ,if the answer is mostly in the midturm index of the word ,when we want to get the representation of the sentence ,we can give the midturm word more attention