PersonalLLMCodingBenchmark
title: Personal LLM Coding Benchmark radar_quadrant: Techniques radar_ring: Assess radar_position: inner created: 2026-05-26 last_updated: 2026-05-26 tags: [llm, evaluation, benchmarking, coding-agents, tooling] source_url: https://blog.ezyang.com/2025/04/why-you-should-maintain-a-personal-llm-coding-benchmark/
A technique for evaluating LLM coding tools (assistants, agents, models) using a personally-maintained suite of real coding tasks, as described by Edward Yang on ezyang's blog in 2025. The approach addresses the gap between marketing benchmarks and individual developer needs.
Public benchmarks (HumanEval, SWE-bench, etc.) measure capabilities on standardised tasks that may not reflect the specific problems a developer encounters. Models are increasingly trained to perform well on known benchmarks, reducing their signal value. Evaluation by intuition ("it feels better") accumulates bias and is difficult to reproduce.
Maintain a personal suite of coding tasks drawn from real work: problems you actually needed to solve, edge cases that tripped up a previous tool, or representative samples from recurring workflows. For each new model or tool:
- Run every task in the suite without modification.
- Score outputs using a consistent rubric (correctness, number of iterations needed, code quality).
- Record results alongside the model version and date.
The benchmark evolves as your work changes — add tasks when you encounter a new failure mode, retire tasks that no longer reflect your actual workflows.
Tasks should be specific enough to have a clear correct answer but not so trivial that all models pass easily. The article recommends avoiding tasks that appear in public training corpora and favouring tasks from internal codebases or domain-specific problems where model training is sparse.
Complements LLM Evaluation Methodology (which covers evaluation design for LLM-powered products) — this technique applies the same discipline to evaluating LLM tools for personal adoption. Also complements Building Effective Agents (Anthropic) by providing a feedback mechanism for agent capability assessment.
Placed in Techniques / Assess / inner. Zero tooling cost — a spreadsheet and a test script suffice. Inner position reflects direct applicability to anyone making tool adoption decisions across Claude, GPT-4, Gemini, or local models. Trial gate: a suite of at least 10 real tasks run against two competing models with results recorded and a tool preference justified by the scores rather than intuition.
