Twenty Questions for Code

Setup

python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

Or use:

bash scripts/setup_env.sh --venv
# or
bash scripts/setup_env.sh --conda

Environment Variables

Create your local env file:

cp .env.example .env

Required in .env:

• CODEGEN_API_KEY
• CODEGEN_MODEL

Optional:

• CODEGEN_BASE_URL
• CODEGEN_TESTGEN_API_KEY
• CODEGEN_TESTGEN_BASE_URL
• CODEGEN_TESTGEN_MODEL

Prepare Datasets

python scripts/prepare_mbpp_variants.py --output data/mbpp_variants.jsonl
python scripts/prepare_humaneval_variants.py --output data/humaneval_variants.jsonl

Run Experiments

MBPP scored on MBPP+:

python scripts/run_experiment.py --config configs/mvp_mbpp.yaml

HumanEval scored on HumanEval+:

python scripts/run_experiment.py --config configs/mvp_humaneval.yaml --enable-humanevalplus

Gemini wrapper scored on HumanEval+:

ENABLE_HUMANEVALPLUS=1 bash scripts/run_google_humaneval_oneshot.sh 20

Summarize Results

python scripts/summarize_results.py \
  --results results/latest.jsonl \
  --output-dir results

Then look at:

• results/summary_mbppplus_pass_at_1.csv
• results/summary_humanevalplus_pass_at_1.csv