Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Two bugs in evaluation scripts #38

Closed
bxclib2 opened this issue Aug 1, 2019 · 1 comment
Closed

Two bugs in evaluation scripts #38

bxclib2 opened this issue Aug 1, 2019 · 1 comment

Comments

@bxclib2
Copy link

bxclib2 commented Aug 1, 2019

Hi

In the evaluation.py

in line 495:

    scores[level]['exec'] = 0  

should be:

    scores[level]['exec'] = 0.  

and in line 549:

should add a line:

            scores["all"]['exec'] += 1

These bugs gives wrong and weird results when evaluate the exec accuracy. Could you fix them? Thanks a lot!

Also, I noticed that even I copy exactly the gold query, the exec accuracy is not 100. Could you check and find the errors??
easy medium hard extra all
count 250 440 174 170 1034
===================== EXECUTION ACCURACY =====================
execution 1.000 0.995 1.000 1.000 0.998

====================== EXACT MATCHING ACCURACY =====================
exact match 1.000 1.000 1.000 1.000 1.000

@taoyds
Copy link
Owner

taoyds commented Jun 8, 2020

Thanks! already corrected in March.

@taoyds taoyds closed this as completed Jun 8, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants