New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Unable to reproduce the results of advGLUE #22
Comments
Hi, We use zero-shot |
Thanks for the reply. But during testing, I found that for each piece of data, the model tended to choose the first option. Have you ever encountered this problem?
|
Hi, Thanks for your careful observation. We did not notice this when we were running Llama2-7b (maybe it does exist). It may come from the position bias of LLMs. How about trying other LLMs to see whether there is such bias? We will check the original results of it and respond to you as soon as possible. |
Thanks for the timely reply, would you please open source the code for obtaining model output? It seems that you use
|
How many shots do you use to test advGLUE?
The text was updated successfully, but these errors were encountered: