Are custom prompts allowed?

[lihaoling]

Can we use customized prompt. e.g. like Magicoder when testing HumanEval, the promtp will appear in the INSTRUCTION and RESPONSE respectively.

 instruction = f"""Write a solution to the following problem:
```python
{prompt}
```"""
response_prefix = f"""```python
{prompt}"""

https://github.com/ise-uiuc/magicoder/blob/main/experiments/text2code.py#L60

Also, can we extract the docstring part of task_prompt from the complete subset of BigCodeBench and use it as the prompt for the INSTRUCTION and task_prompt as the prompt for the RESPONSE?

[terryyz]

Hi @lihaoling,

Regarding the customized prompt, please refer to #57.

For the docstring extraction, we provide the doc_struct column in BigCodeBench Data and BigCodeBench-Hard Data. Hope it helps :-)

[lihaoling]

Thank you very much for your response timely!

I have understood the rules for setting up a customized prompt. How can we make sure that our test results match yours after making a Model Evaluation Request, can we provide you with a custom prompt or can we submit you a pre-generated solution?

[terryyz]

Hi @lihaoling,

I just added a Result Submission section. Please send us your outputs and results if you'd like to make your models a part of our leaderboard.

[lihaoling]

Thank you very much for your thoughtful help, BigCodeBench is really a great job!!!