Is attention normalization method right?

[candlewill]

@r9y9 To my understanding, x = x * (s * math.sqrt(1.0 / s)) == x = x * (math.sqrt(s)). Is this right?

If it is right, why we need to multiply x with math.sqrt(s) instead of divided by math.sqrt(s) ?

deepvoice3_pytorch/deepvoice3_pytorch/deepvoice3.py

Line 172 in ec58290

x = x * (s * math.sqrt(1.0 / s))

[r9y9]

I think you are right. To note, this is because the code was adapted from fairseq https://github.com/facebookresearch/fairseq-py/blob/9430544a3b95dbb2e3b86c303776e37f18188ad3/fairseq/models/fconv.py#L139

They use the notation m * sqrt(1/m) in their paper (see sec. 3.4 of https://arxiv.org/pdf/1705.03122.pdf) and I guess this is why the code as is.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.