Maybe scale is wrong

[denadai2]

memorizing-transformers-pytorch/memorizing_transformers_pytorch/memorizing_transformers_pytorch.py

Line 237 in 83fa147

sim = einsum('b h i d, b j d -> b h i j', q, k) * scale

Shouldn't this be (1-scale)?

[lucidrains]

ohh no, that is actually the learned temperature from a variant of attention (cosine similarity attention) https://github.com/lucidrains/x-transformers#query-key-normalization the temperature is in log space, exponentiated here https://github.com/lucidrains/memorizing-transformers-pytorch/blob/main/memorizing_transformers_pytorch/memorizing_transformers_pytorch.py#L235

[lucidrains]

@denadai2 ohh if you were looking for the sigmoid gating, i removed that, since it was not working well for me and another researcher (thought that was one of the weak parts of the paper). i went with the other researcher's suggestion of attending across the similarities, local and distant (softmax across the attention logits concatted)

[denadai2]

thanks for the prompt answer! I saw it now :)

btw this increases the complexity I'd say... it makes sense though