The wrong imp of the inner-product operation

[XiaLiPKU]

In Equation 2 of the paper, the query and the key are fed into inner-product operation, instead of point multiplication.

So the follow line

Stand-Alone-Self-Attention/attention.py

Line 48 in e0a168e

out = q_out * k_out

should be
out = (q_out * k_out).sum(dim=2)

[20171130]

I found the same problem. It seems the implementation in the code is equivalent to having #attention heads = #embed dimensions.

[ifeherva]

@XiaLiPKU How would that modify line 49 and 50?

[canaltin]

@20171130 That was also my first opinion, but then there is an inconsistency with "groups" definition (to replicate the "attention heads") throughout the paper & the code.

Anyway, your alternative implementation helped me to understand the general concepts:
https://github.com/20171130/AttentionLite/blob/master/model.py