How is KodCode-R1-SFT-V1 validated?

[cizhenshi]

Thankf for your work!
I found that in the KodCode-R1-SFT-V1 dataset, the function names written by the model in the results where r1-correctness is true do not always match those in test_info. In this case, how is the code validation performed using pytest?

[zhangchen-xu]

Hi,

Thank you for your question. By default, we enable the solution function name replacement in order to keep more data.

Please refer to the code for details:

kodcode/pipeline/step3.3.3_gen_unit_tests_sft.py

Line 81 in b2042ed

if solution_func_match and args.replace_function_name and len(re.findall(r'def\s+\w+\s*\(', solution_code)) == 1:

[cizhenshi]

Thank you for your response. However, I noticed that in the code, only the function name in test_info[0] is replaced. In the actual data, there are cases that include multiple test_infos, and the entry function is not always test_info[0]. How is this situation handled?
such as
question:
Write a function that receives two integers, start and end, and returns the sum of all prime numbers in the inclusive range [start, end]. If there are no prime numbers in the given range, the function should return 0.
test_info:
[{'docstring': 'Returns True if n is a prime number, otherwise False.', 'function_declaration': 'def is_prime(n):', 'function_name': 'is_prime', 'parameter_list': '(n)'}, {'docstring': 'Returns the sum of all prime numbers in the inclusive range [start, end].\nIf there are no primes in the range, returns 0.', 'function_declaration': 'def sum_of_primes(start, end):', 'function_name': 'sum_of_primes', 'parameter_list': '(start, end)'}]

[zhangchen-xu]

Oh when doing replacement, we only consider solutions with the length of test_info = 1.

Detail codes are here:

kodcode/pipeline/step3.3.3_gen_unit_tests_sft.py

Line 58 in b2042ed

if len(generated_data["metadata"]["test_info"]) == 1:

[cizhenshi]

Does this mean that part of the low pass rate results is due to the presence of multiple test_infos? So, should I discard this part of the data when using it?

[zhangchen-xu]

Most of the test info length is 1. Yes, I suggest discard entries with multiple test_infos if you are focusing on function implementation. We also do the similar filtering when we were selecting our 50K data, following common practices in https://arxiv.org/abs/2502.01718.

Usually, if there are more than one test_infos for a certain question, the question is quite challenging lol.

[cizhenshi]

I understand now. Thank you very much!