code_repair task evaluation

[scorpio-nova]

Dear Authors,
Thanks for the great work! I'm planning to do some work based on this dataset, and encounter a small problem:
Following README, I've run the code_repair inference for several open-source LLMs, and the results are saved to CodeScope/code_repair/inference/result/code_repair_eval_{model_name}.jsonl by default.
I'm confused about the next step to take.

1. The jsonl file looks like this
   
   If I'm not mistaken, source_code is the code for LLM to debug, code_repairing_0 is the output of the LLM, with code and some explanations.
2. However, the next step shows that the Evaluator seems to need an input jsonl in a different format?
   

The question is how to change the jsonl in 1 to the required input format of 2, Specifically:

1. Should the "source_code" in 2 replaced by the code extracted from the model's output, are there some code to enable this conversion?
2. for "lang_cluster", "lang" etc. should I replace the value to "{model_name}" in the README? Or it's just a placeholder and I just keep the original value in the code_repair_eval_{model_name}.jsonl without changing anything?

Thanks for your help!

[yunzhel2]

Hi,

Thanks for your interests in our work. Exactly! Your thoughts about the input and output are totally correct.

In step 2, you will use the execute_engine to run your code. It is designed for testing several different tasks(program_synthesis, repair, translation). So the meaning is quite different. In repair task, the "source_code" should be model's output. You could write a simple script for changing it. Btw, the model sometimes will create some structural information which is not related to the question, e.g., code block syntax. You may need to make a quick check in the script.

For other execution parameters, keeping the original value in code_repair_eval_{model_name}.jsonl is okay.

Hopefully these could solve your questions. Feel free to let me know if you have any further questions.

-Yunzhe

[scorpio-nova]

Hi,

Thanks for your interests in our work. Exactly! Your thoughts about the input and output are totally correct.

In step 2, you will use the execute_engine to run your code. It is designed for testing several different tasks(program_synthesis, repair, translation). So the meaning is quite different. In repair task, the "source_code" should be model's output. You could write a simple script for changing it. Btw, the model sometimes will create some structural information which is not related to the question, e.g., code block syntax. You may need to make a quick check in the script.

For other execution parameters, keeping the original value in code_repair_eval_{model_name}.jsonl is okay.

Hopefully these could solve your questions. Feel free to let me know if you have any further questions.

-Yunzhe

Thanks for your timely response! I can run it now :)