Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Positional Encoding #32

Closed
jaseleephd opened this issue Nov 11, 2017 · 1 comment
Closed

Positional Encoding #32

jaseleephd opened this issue Nov 11, 2017 · 1 comment

Comments

@jaseleephd
Copy link

jaseleephd commented Nov 11, 2017 •
edited
Loading

In position_encoding_init, shouldn't it be

[pos / np.power(10000, (i//2)*2 / d_pos_vec ) for i in range(d_pos_vec)]

instead of

[pos / np.power(10000, 2*i/d_pos_vec) for i in range(d_pos_vec)]

In the original formulation, for positions 2i and 2i+1, the power should be 2i / d_model.
jadore801120 added a commit that referenced this issue Nov 19, 2017
@jadore801120
fix position encoding in #32
7fa8c63
@jadore801120
Copy link
Owner

Hi @jasonleeinf

Thank you so much to point out this mistake!
This part has been fixed in 7fa8c63. Please take a look.
Thanks!

Yu-Hsiang

r9y9 added a commit to r9y9/deepvoice3_pytorch that referenced this issue Dec 15, 2017
@r9y9
Fix bug for sinsoidal encoding
13772d8 
jadore801120/attention-is-all-you-need-pytorch#32
r9y9 added a commit to r9y9/deepvoice3_pytorch that referenced this issue Dec 18, 2017
@r9y9
Fix bug for sinsoidal encoding
a2eb566 
jadore801120/attention-is-all-you-need-pytorch#32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@jadore801120 @jaseleephd