Evaluation for HumanEval(+) and MBPP(+)

Update

2024-04-30: we have released our first finetuned 6.7B chat model aigcode/AIGCodeGeek-DS-6.7B and provided code for EvalPlus evaluation in this repo.

---

This repository contains scripts and code to run EvalPlus benchmarks on AIGCodeGeek series models.

EvalPlus includes HumanEval(+) and MBPP(+) datasets for evaluating code completion preformance.

1. Generations and Results

generations/ stores the generation files and results/ gives the corresponding eval results:

generations
├── humaneval
│   └── aigcode--AIGCodeGeek-DS-6.7B_temp_0.0
├── mbpp
    └── aigcode--AIGCodeGeek-DS-6.7B_temp_0.0

results
├── humaneval
│   └── aigcodev1_temp_0.0.txt
└── mbpp
    └── aigcodev1_temp_0.0.txt

Both Base and Plus results are reported:

Model	HumanEval	HumanEval+	MBPP	MBPP+
AIGCodeGeek-DS-6.7B	82.3	76.2	77.7	64.4

2. Environment

pip install evalplus --upgrade
pip install -r requirements.txt

3. Evalutaion

The evaluation is done with a single A100 GPU. Here are scripts:

bash run_humaneval.sh
bash run_mbpp.sh