-
Notifications
You must be signed in to change notification settings - Fork 132
held out benchmark too optimistic
A test split that comes out of the same process as the training data is not held out enough, even when whole topics are kept aside. Our held-out split for jevos excluded three business policies and three workflows entirely, and the model scored 0.855 on it. On 999 questions written from scratch after training, by a different process, it scored 0.757. The ten points between the two are the part of the first number that measured familiarity with our own way of writing questions, not the ability to answer them.
This matters beyond our model. Anyone fine-tuning a model on generated or templated data and testing on a split of the same data is likely to be reporting the optimistic number.
This page is the four measurements from most familiar to least, why excluding topics was not enough, and how we build tests now.
| Test | What is new to the model | Accuracy |
|---|---|---|
| development split, 12,320 questions | nothing but the specific texts | 0.915 |
| held-out split, 27,274 questions | three policies and three workflows never seen | 0.855 |
| 2,000 questions on the held-out policies, answers computed by code | the policies, and a separate question set | 0.810 |
| 999 questions written after training | everything: texts, questions, phrasing, author | 0.757 |
Each step away from the training data costs accuracy, and the biggest step is the last one, where nothing was produced by the same pipeline. The held-out split feels like a hard test, since whole domains are missing from training, and it still sits closer to the development number than to the independent one.
One caveat, stated so it can be weighed: the first two rows were measured on the q8_0 build
and the last on q4_k_m. We have not run both builds on the same set for this page, so some part
of the gap could be quantization. The ordering of the middle rows, and the size of the last step,
do not depend on it.
Holding out a policy removes its rules from training. It does not remove:
- the question templates. A question about an unseen policy can still be phrased the way a thousand training questions were phrased.
- the text formats. A ticket, an email or a JSON record produced the same way looks like every other ticket, email and record in training, whatever it is about.
- the answer distribution. The balance of yes and no, and what a hard case looks like, come from the same place.
A model can learn all three, and a test that shares them rewards it for doing so. The 999 questions share none of them, and on those the model's weaknesses show up that the held-out split had hidden: arithmetic at 0.584 and a lean toward yes that the held-out calibration (0.009 error on its natural questions) gave no hint of. Both are measured on why a small LLM says yes and small LLMs and arithmetic.
Written after training, by a different process. The 999 questions were written from scratch once the model was finished: 10 scenarios, 10 texts per scenario of 40 to 150 words, 10 questions per text, exactly half of them yes. None of the text came from the pipeline that made the training data.
Labelled by kind. Each question carries the reasoning it needs: stated fact, tone, paraphrase, intent, negation, not stated, rule, number, date, arithmetic. A single accuracy hides where a model fails; ten accuracies say what to fix and what to route around.
Never tuned on. The set is used to report, not to choose. The moment a threshold, a prompt or a checkpoint is picked by looking at it, it becomes a development set and a new test is needed. We do not publish its questions for the same reason.
A cheap second test with exact answers. Hand-written questions can be mislabelled, so we also generate 1,000 questions from templates with answers computed by code. It is not independent in the same way, since templates have a style of their own, but it has no labelling errors, it can be regenerated at any size, and on the kinds that matter it agreed with the hand-written set on arithmetic and dates within two points.
For jevos, the number to plan with is the independent one, 0.757 overall, together with its breakdown: above 0.84 on everything that is reading, 0.58 to 0.72 on everything that is computing or applying a rule. The README reports the 2,000-question comparison, 0.810, because the same questions were put to Jev and Laya and the comparison is fair; this page is the context for reading it.
Why is my model worse in production than on the test set? If the test set was cut from the same data as the training set, it shares templates, formats and style with it. Ours overstated accuracy by about ten points.
Is holding out whole topics enough? Not in our measurement: excluding three policies and three workflows still gave 0.855, against 0.757 on independent questions.
How big should an independent test be? Ours has 999 questions, enough to report accuracy per kind of question with about 30 to 220 questions per kind.
Should I publish my test questions? Not if you want to keep using them: published questions end up in someone's training data.
What is a good cheap check? Questions generated from templates with answers computed by code. They have no labelling errors and reproduced our hardest cases.
See also: why a small LLM says yes when the answer is no, small LLMs and arithmetic in yes/no questions and jevos vs Jev vs Laya.
- All four accuracies are our own measurements of the released jevos: development and held-out
splits on
jevos-q8_0, the 2,000-question comparison as reported in the jev README, and the 999-question set onjevos-q4_k_m.
From the notes of jev, a yes/no decision model that runs on a laptop CPU. We report the lower number because it is the one your application will see.
- Ask a local LLM a yes/no question and get P(yes)
- Zero-shot text classification with yes/no questions
- LLM policy decisions: put the rule in the question
- LLM as a judge on a CPU
- Why a small LLM says yes when the answer is no
- Small LLMs and arithmetic in yes/no questions
- Our held-out benchmark said 0.855, new questions said 0.757
- jevos vs Jev vs Laya for yes/no decisions
- An open-source alternative to Jev for yes/no decisions
- jevos vs the OpenAI API for yes/no classification
- jevos vs Ollama for yes/no decisions
- jevos vs bart-large-mnli for zero-shot classification
- A yes/no LLM vs a fine-tuned BERT classifier
- jevos vs SetFit: zero-shot vs few-shot classification
- jevos vs Llama Guard for content safety checks
- jev serve vs llama.cpp server for classification
- jevos vs LM Studio: a decision server, not a chat app
- Local vs hosted LLM decisions: latency, cost, privacy
- A yes/no LLM vs a business rules engine
- LLM decisions vs keyword rules and regex
- The fastest AI model for yes/no decisions
- What makes a local LLM fast on a CPU
- Why one forward pass beats generating an answer
- Prefill vs decode: where LLM latency comes from
- Why LLM latency grows with the length of the text
- Why a hosted LLM API cannot answer in 50 ms
- Many questions about one text: why the extra ones are cheap
- CPU or GPU for a small LLM
- Latency budgets: where a 200 ms model fits
- Measuring LLM latency: median, p90 and warm-up
- Q4_K_M vs Q8_0: speed and size for a small model
- Throughput vs latency for a decision server
- What P(yes) means, and what it does not
- LLM calibration explained with yes/no answers
- Expected calibration error (ECE), explained
- Temperature scaling for LLM probabilities
- Platt scaling for a yes/no model
- Reading a reliability diagram
- How to choose a threshold for P(yes)
- Thresholds when a wrong yes costs more than a wrong no
- Human in the loop AI with a review band
- Precision and recall at a P(yes) threshold
- Base rates: why a 0.9 yes can still be wrong often
- Combining yes/no answers with AND, OR and NOT
- Logits, log-odds and P(yes)
- LLM confidence scores: probabilities vs self-reports
- How to write yes/no questions an LLM answers well
- Negation in yes/no questions for an LLM
- One condition per question: splitting compound questions
- Ask whether the text says it at all
- Scores as yes/no thresholds: is it at least high?
- Sending JSON as the text: designing the state
- Why wording changes an LLM's answer, and how to test it
- Mainly about: questions for messages with several topics
- Yes/no questions about tone and emotion
- Asking about intent: what does the writer want?
- Yes/no questions about long documents
- Using an English-only LLM with other languages
- Content moderation with a local LLM
- A Discord moderation bot with a local LLM
- Spam detection with yes/no questions
- Review moderation with a local LLM
- Email triage with a local LLM
- Support ticket routing with yes/no questions
- Urgency detection in customer messages
- Sentiment analysis with yes/no questions
- Intent detection with a local LLM
- Lead qualification with yes/no questions
- Fraud case triage with a local LLM
- Phishing email screening with a local LLM
- Log and alert triage with a local LLM
- Checking text for personal data with yes/no questions
- Prompt injection screening with a small model
- Document classification with a local LLM
- Product categorization with yes/no questions
- Contract clause detection with a local LLM
- Refund request triage with a local LLM
- Detecting cancellation intent in customer messages
- RAG evaluation with yes/no questions
- RAG faithfulness check with a local LLM
- Hallucination detection with a local LLM
- LLM regression tests in CI with yes/no checks
- Rubric design for an LLM judge
- Pairwise comparison with a yes/no judge
- LLM judge bias and how to control it
- Evaluation metrics for yes/no classifiers
- Building a yes/no test set for your own data
- Accuracy by kind of question: why one number hides failures
- Generating test questions with answers computed by code
- Benchmark contamination and truly held-out tests
- An LLM router with yes/no questions
- A model cascade: small model first, large model on doubt
- Semantic routing vs yes/no questions
- Gating AI agent tool calls with yes/no checks
- AI agent guardrails with yes/no questions
- Stop conditions for AI agents
- Logging LLM decisions for audit
- Reducing LLM cost with local yes/no decisions
- Replacing chat LLM calls with yes/no questions
- Structured output vs a probability
- A Python client for local LLM decisions
- Calling a local LLM decision server from JavaScript
- Local LLM yes/no decisions in n8n
- A Slack bot that uses local LLM decisions
- Home Assistant automations with local LLM decisions
- A LangChain tool for local yes/no decisions
- Batch decisions from files with jev decide
- Running LLM yes/no checks in GitHub Actions
- Securing a local LLM server with an API key
- curl examples for a local LLM decision API
- Self-hosted AI for decisions
- A private LLM for text classification
- On-premise LLM for business decisions
- GDPR and automated decision-making with an LLM
- Offline AI for decisions: no network needed
- Edge AI decisions on a CPU
- Run an LLM locally without a GPU
- Small language models explained
- When a small model is enough, and when it is not
- An LLM on a laptop: what it can do in real time
- What is GGUF, for someone deploying a classifier
- GGUF quantization types explained: Q4_K_M, Q8_0 and others
- GGUF vs safetensors
- llama.cpp vs Ollama for a classification service
- llama-cpp-python vs calling llama.cpp through ctypes
- llama.cpp on Windows without compiling
- Running llama.cpp CPU only
- Using llama.cpp prebuilt binaries instead of building