Code benchmarks

[TreeinRandomForest]

While we are starting with SWE-bench Verified, we should explore other python benchmarks. In increasing order of difficulty:

• HumanEval: ~160 hand-crafted python functions. Difficulty - Easy
• ODEX: Open-domain execution-based python. Library API use. Difficulty - Easy-Medium
• BigCodeBench: ~1700 tasks using composition of the standard library + third-party libraries. Difficulty - Medium
• LiveCodeBench: Competitive programming. Contamination resistant. Difficulty: Medium-Very Hard
• SWE-bench Verified: 500 human-validated GitHub issues from real python repos. Difficulty: Hard

Warning: See paper for issues with memorization.

Agentic coverage:

• SWT-Bench: Test generation for a given codebase. Includes categories like Test Generation, Test Repair, Coverage Improvement.
• Terminal-Bench 2.0: Tasks in a terminal/shell environment.
• FeatureBench: Feature request implementation and not just bug fixes.

Python DS code:

• DS-1000: ~1000 data science problems in numpy, pandas, matplotlib, sklearn, pytorch, scipy.
• MLE-bench: Kaggle competitions benchmark

Contamination resistance:

• EvoEval: Semantic perturbations of HumanEval problems.

Possible curriculum for RL:

1. Easy HumanEval -> EvoEval -> ODEX
2. Medium BigCodeBench -> DS-1000 -> LiveCodeBench (easy/medium)
3. Hard SWE-bench Verified -> SWT-Bench -> Terminal Bench 2.0
4. Very Hard MLE-Bench -> LiveCodeBench (hard) -> FeatureBench

Important point about environment

Most of these benchmarks have examples that fit in the context window. For practical use, once codebases get large enough, retrieval plays an important role. Might need custom harness to reproduce this e.g. cap the number of files that can be read for any instance. Don't worry about this now!

[TreeinRandomForest]

@MichaelClifford @lmzuccarelli Found many more benchmarks with different difficulty levels and different tasks.

[TreeinRandomForest]

HumanEval (1 dataset), EvoEval (5 datasets - tooluse, subtle, combine, creative difficult), BigCodeBench have essentially the same format. Each data element consists of:

• Task ID: a unique string identifying each task.
• prompt: a set of function names (def funcname) with type signatures and a docstring describing the task. Task is to fill in function definitions.
• canonical_solution: function definitions.
• entry_point: the function being called.
• test: test functions consisting of asserts.

HumanEval is the easiest. EvoEval tooluse and subtle datasets are easy enough to act as diagnostic benchmarks. EvoEval combine is generally the hardest. EvoEval creative and difficult (and possibly creative) are good candidates for RL. BigCodeBench has two prompts - complete_prompt (docstring based like HumanEval and EvoEval) and instruct_eval (natural language description that makes the task harder). BigCodeBench has its own harness.