AIEvalsFAQ
title: Frequently Asked Questions About AI Evals radar_quadrant: Techniques radar_ring: Assess radar_position: inner created: 2026-05-26 last_updated: 2026-05-26 tags: [llm, evals, evaluation, testing, hamel] source_url: https://hamel.dev/blog/posts/evals-faq/
A practitioner FAQ on LLM evaluation by Hamel Husain, published on hamel.dev. Addresses the most common points of confusion for teams starting to write evals for LLM-powered applications.
What counts as an eval? Any automated check on LLM output quality — from a simple assertion ("the response is not empty") to a complex LLM-as-judge scoring pipeline. The threshold is low: if it runs automatically and produces a signal, it is an eval.
When to start? Before you have a production system. Evals written against early prototypes catch regressions as the system evolves. Starting late means shipping regressions silently.
How many do you need? Fewer than you think to start. Ten representative examples covering failure modes beat one hundred shallow examples. Grow the suite as new failures are discovered in production.
Unit evals vs. system evals. Unit evals test a single LLM call in isolation (a prompt template, a retrieval step). System evals test the full pipeline end-to-end. Both are necessary; unit evals catch regressions faster, system evals catch emergent failures.
No ground truth? Use LLM-as-judge: a separate model scores the output against criteria. Validate the judge's scoring against human ratings on a calibration set before trusting it at scale.
The most common mistake? Waiting until the system is "done" before writing evals. Evals are not a QA step at the end; they are a design tool throughout.
Complements LLM Evaluation Methodology (which covers the full framework: SMART criteria, automated grading, LLM-as-judge) — this FAQ is the on-ramp that answers the "where do I actually start?" question practitioners get stuck on. Also complements Personal LLM Coding Benchmark (evaluating LLM tools) and Building Effective Agents (Anthropic) (agent system design).
Placed in Techniques / Assess / inner. The FAQ format makes it more immediately actionable than most eval content — each answer is a concrete unblocking for a real practitioner question. Inner position reflects zero tooling cost and direct applicability to any team building LLM-powered features. Trial gate: eval suite of at least 10 examples covering real failure modes, running automatically on every prompt or pipeline change, with at least one LLM-as-judge scorer calibrated against human ratings.
