Buggy test set?

[alibrahimzada]

I am working with AVATAR and tried to extract the test set with its test cases. I was able to extract 252 instances with test cases from codeforces and atcoder. I am facing some issues with test cases where the expected_output or test_input has ... at the end of it. I believe when downloading and preparing the test set, some inputs/outputs are getting truncated and ... is added at the end of test input/output. Moreover, there are test cases where the code is expecting 2 inputs but there is only one input in the test case. So as a result, the program hangs up waiting for the second input. I get these issues after doing the following:

1. Downloading the dataset by executing bash download.sh and prepare.sh in data
2. Downloading test cases by executing bash download.sh and bash prepare.sh in test_cases
3. The created atcoder_id2tests_filtered.jsonl and codeforces_id2tests_filtered.jsonl has avatar IDs but the inputs and outputs fields are empty ({"avatar_id": "codeforces_313_B", "inputs": [], "outputs": []}).
4. I matched the keys available in filtered jsonl files to non-filtered ones and extracted all unit tests for each example. For instance, this is the one for codeforces_313_B: {"avatar_id": "codeforces_313_B", "inputs": ["313_B/samples/10_input.txt", "313_B/samples/31_input.txt", "313_B/samples/25_input.txt", "313_B/samples/2_input.txt", "313_B/samples/28_input.txt", "313_B/samples/37_input.txt", "313_B/samples/23_input.txt", "313_B/samples/9_input.txt", "313_B/samples/16_input.txt", "313_B/samples/4_input.txt", "313_B/samples/11_input.txt", "313_B/samples/24_input.txt", "313_B/samples/30_input.txt", "313_B/samples/3_input.txt", "313_B/samples/29_input.txt", "313_B/samples/22_input.txt", "313_B/samples/8_input.txt", "313_B/samples/36_input.txt", "313_B/samples/17_input.txt", "313_B/samples/5_input.txt", "313_B/samples/33_input.txt", "313_B/samples/27_input.txt", "313_B/samples/12_input.txt", "313_B/samples/14_input.txt", "313_B/samples/35_input.txt", "313_B/samples/21_input.txt", "313_B/samples/19_input.txt", "313_B/samples/6_input.txt", "313_B/samples/26_input.txt", "313_B/samples/32_input.txt", "313_B/samples/13_input.txt", "313_B/samples/1_input.txt", "313_B/samples/15_input.txt", "313_B/samples/20_input.txt", "313_B/samples/34_input.txt", "313_B/samples/18_input.txt", "313_B/samples/7_input.txt"], "outputs": ["313_B/samples/10_output.txt", "313_B/samples/31_output.txt", "313_B/samples/25_output.txt", "313_B/samples/2_output.txt", "313_B/samples/28_output.txt", "313_B/samples/37_output.txt", "313_B/samples/23_output.txt", "313_B/samples/9_output.txt", "313_B/samples/16_output.txt", "313_B/samples/4_output.txt", "313_B/samples/11_output.txt", "313_B/samples/24_output.txt", "313_B/samples/30_output.txt", "313_B/samples/3_output.txt", "313_B/samples/29_output.txt", "313_B/samples/22_output.txt", "313_B/samples/8_output.txt", "313_B/samples/36_output.txt", "313_B/samples/17_output.txt", "313_B/samples/5_output.txt", "313_B/samples/33_output.txt", "313_B/samples/27_output.txt", "313_B/samples/12_output.txt", "313_B/samples/14_output.txt", "313_B/samples/35_output.txt", "313_B/samples/21_output.txt", "313_B/samples/19_output.txt", "313_B/samples/6_output.txt", "313_B/samples/26_output.txt", "313_B/samples/32_output.txt", "313_B/samples/13_output.txt", "313_B/samples/1_output.txt", "313_B/samples/15_output.txt", "313_B/samples/20_output.txt", "313_B/samples/34_output.txt", "313_B/samples/18_output.txt", "313_B/samples/7_output.txt"]}
5. Assuming these inputs and outputs are correct, I compiled and executed codeforces_313_B.java with the input provided in 313_B/samples/10_input.txt but since its one line, the program hangs up and waits for another input. However, no more input is available in 10_input.txt.
6. I believe ... can be part of input, but for some test cases the program parses it and tries to change everything to int, but it throws an exception when converting ... to integer.
7. I believe filtering should take care of this issue, however my filtered.jsonl files has no inputs/outputs. If the authors have jsonl files, it would be great if they can share it because I could not reproduce them.

Thanks.

[alibrahimzada]

Closing this issue because I debugged filter.py and solve the issue. Sorry for the inconvenience.

[abaveja313]

Would you mind sending your fix @alibrahimzada