I find that when using test.py to evaluate shine, no_metanet, and only_question, only inputs in no_metanet and only_question group include the system prompt, which may result in unfairness.
input in no_metanet:
"input": "system\nYou are a concise assistant. Output only the final answer, in a few words, as short as possible. No explanations. Do not output anything else.\nuser\nReference:\nPrivate schooling in the United States has been debated by educators, lawmakers and parents, since the beginnings of compulsory education in Massachusetts in 1852. The Supreme Court precedent appears to favor educational choice, so long as states may set standards for educational accomplishment. Some of the most relevant Supreme Court case law on this is as follows: Runyon v. McCrary, 427 U.S. 160 (1976); Wisconsin v. Yoder, 406 U.S. 205 (1972); Pierce v. Society of Sisters, 268 U.S. 510 (1925); Meyer v. Nebraska, 262 U.S. 390 (1923).\n\nBased on the reference, answer this question:\nIn what year did Massachusetts first require children to be educated in schools?\nassistant\n\n\n\n\n",
input in shine:
"input": "user\nIn what year did Massachusetts first require children to be educated in schools?\nassistant\n",
I find that when using test.py to evaluate shine, no_metanet, and only_question, only inputs in no_metanet and only_question group include the system prompt, which may result in unfairness.
input in no_metanet:
"input": "system\nYou are a concise assistant. Output only the final answer, in a few words, as short as possible. No explanations. Do not output anything else.\nuser\nReference:\nPrivate schooling in the United States has been debated by educators, lawmakers and parents, since the beginnings of compulsory education in Massachusetts in 1852. The Supreme Court precedent appears to favor educational choice, so long as states may set standards for educational accomplishment. Some of the most relevant Supreme Court case law on this is as follows: Runyon v. McCrary, 427 U.S. 160 (1976); Wisconsin v. Yoder, 406 U.S. 205 (1972); Pierce v. Society of Sisters, 268 U.S. 510 (1925); Meyer v. Nebraska, 262 U.S. 390 (1923).\n\nBased on the reference, answer this question:\nIn what year did Massachusetts first require children to be educated in schools?\nassistant\n\n\n\n\n",
input in shine:
"input": "user\nIn what year did Massachusetts first require children to be educated in schools?\nassistant\n",