Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

How are A1,A2, A3 scores calculated from r1,r2,r3 test results? #2

Closed
ishanisri opened this issue May 6, 2021 · 2 comments
Closed

How are A1,A2, A3 scores calculated from r1,r2,r3 test results? #2

ishanisri opened this issue May 6, 2021 · 2 comments

Comments

@ishanisri
Copy link

The results of evaluation show here in the README file vs the ones in the paper are based on different metrics. The paper shows A1,A2,A3 scores along with accuracy on adv-MNLi, adv-SNLi etc, whereas running the evaluation here shows us accuracy roundwise of both tests and dev data. Is there any formula to calculate the A1,A2,A3 scores from the test scores mentioned in the repo?

Thanks in advance for the help :)
@boxin-wbx
Copy link
Member

You may find A1, A2, A3 test scores in the log, which are respectively anli-r1-test, anli-r2-test, anli-r3-test.

@ishanisri
Copy link
Author

Ah okay, this helped a lot, thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@boxin-wbx @ishanisri