New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RNN results not reproducible #50

Open
Wronskia opened this Issue Jun 15, 2018 · 2 comments

Comments

Projects
None yet
3 participants
@Wronskia
Copy link

Wronskia commented Jun 15, 2018

Hello,
Thanks for open sourcing the code.
After your commit:
2734eb2

I get 63.26 in ppl and not the 55.6 stated in the paper. However before this commit I get 55.6. Is there something I am missing?

Thanks

@hyhieu

This comment has been minimized.

Copy link
Collaborator

hyhieu commented Jun 17, 2018

Hi,

Thanks for your interest. Commit 2734eb2 fixed a bug in the evaluation process. After the fix, we had to further tune the model's hyper-parameters to reach a good performance. The best number we could reach was 56.3. We have updated the paper and will soon role out a commit to fix the bug in the code. We apologize for the mistake.

@liamcli

This comment has been minimized.

Copy link

liamcli commented Oct 16, 2018

Could you update the ptb_final.sh script to use the architecture and hyperparameters you used to get to the 56.3 perplexity reported in the paper? Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment