Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Dramatic performance degradation w/o top-k trick #25

Closed
FrontierBreaker opened this issue Jul 17, 2021 · 3 comments
Closed

Dramatic performance degradation w/o top-k trick #25

FrontierBreaker opened this issue Jul 17, 2021 · 3 comments

Comments

@FrontierBreaker
Copy link

According to my retraining, I got ~84 J&F w/ top-k but ~80 J&F w/o top-k. It seems this trick influences the overall performance greatly. Have you observed such a phenomenon in your exp?
@FrontierBreaker
Copy link
Author

By the way, have you studied the selection of Hyper-param "K" in top-k filtering?

@hkchengrex
Copy link
Owner

Our s03 model drops to 84.1 without top-k. Top-k is a pretty neat trick and we have also applied it to the baseline in all ablation studies. We just tried a few values and picked 20.

@hkchengrex
Copy link
Owner

Please see the updated readme for further help in reproducibility.

@zhouweii234 zhouweii234 mentioned this issue Aug 31, 2021
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@hkchengrex @FrontierBreaker