Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Question about the human eval result, thank you. #76

Open
guotong1988 opened this issue Aug 26, 2023 · 2 comments
Open

Question about the human eval result, thank you. #76

guotong1988 opened this issue Aug 26, 2023 · 2 comments

Comments

@guotong1988
Copy link
Contributor

guotong1988 commented Aug 26, 2023

In 12 line of human eval result
the human result is 4,750.

But in the test.jsonl, the SQL is:

{"phase": 2, "table_id": "2-15062099-1", 
"question": "What is the capacity for the stadium of kadrioru stadium and a current manager of sergei ratnikov?", 
"sql": {"sel": 3, "conds": [[2, 0, "kadrioru stadium"], [4, 0, "sergei ratnikov"]], "agg": 5}}

In the test.tables.jsonl, the table is:

{"header": ["Club", "Location", "Stadium", "Capacity", "Current manager"], 
"page_title": "2008 Meistriliiga", 
"types": ["text", "text", "text", "real", "text"], 
"page_id": 15062099, "id": "2-15062099-1", "section_title": "Clubs and venues", 
"rows": 
[["Flora", "Tallinn", "A. Le Coq Arena", "9,300", "Pasi Rautiainen"], 
["Kalju", "Tallinn", "Hiiu Stadium", "500", "Fredo Getulio"], 
["Levadia", "Tallinn", "Kadrioru Stadium", "4,750", "Igor Prins"], 
["Maag Tammeka", "Tartu", "Tamme Stadium", "2,000", "Sergei Zamogiln\u00f5i"], 
["Sillam\u00e4e Kalev", "Sillam\u00e4e", "Sillam\u00e4e Kalevi Stadium", "2,000", "Vadym Dobizha"], 
["Tallinna Kalev", "Tallinn", "Kalevi Keskstaadion", "12,000", "Aavo Sarap"], 
["Trans", "Narva", "Kreenholmi Stadium", "3,000", "Aleksei Yagudin"],
 ["Tulevik", "Viljandi", "Viljandi linnastaadion", "2,500", "Marko Lelov"], 
["TVMK", "Tallinn", "Kadrioru Stadium", "4,750", "Sergei Ratnikov"], 
["Vaprus", "P\u00e4rnu", "P\u00e4rnu Kalevi Stadium", "1,900", "Ants Kommussaar"]], 
"caption": "Clubs and venues"}

How can the SQL execution result be [4.0]?

@guotong1988
Copy link
Contributor Author

@whwang299 Thank you.

@guotong1988
Copy link
Contributor Author

guotong1988 commented Aug 26, 2023

In 17 line of human eval result

The SQL is wrong. I understand. The SQL do not correspond to the question.

In the test.jsonl, the SQL is:

{"phase": 2, "table_id": "2-12821570-2", 
"question": "What was the highest number of points scored when there was a draw of 2 and when the amount played was 18?", 
"sql": {"sel": 7, "conds": [[3, 0, 2], [2, 2, 18]], "agg": 1}}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant