-
Notifications
You must be signed in to change notification settings - Fork 132
precision and recall at a threshold
Raising the P(yes) threshold usually raises precision, because fewer wrong yeses get through, and lowers recall, because more true yeses fall below the bar; lowering it does the opposite. Precision is the share of your yeses that are right; recall is the share of the real yeses you caught. Which one to favour depends on which mistake hurts: a spam filter that hides real mail needs precision, while a queue that must not miss a refund request needs recall. The same question can serve both, with a different threshold for each action.
There is no threshold that is best in general, only one that is best for a decision. The useful habit is to name the action first ("move to spam", "route to the refund queue", "pay out") and then ask which error that action can afford.
This page is the definitions, a worked example with toy numbers, two use cases that pull in opposite directions, why precision depends on how common yes is, and how to draw the curve from your own answers.
scikit-learn defines them with the counts of true positives (tp), false positives (fp) and false negatives (fn):
- Precision = tp / (tp + fp): of the cases you called yes, how many were yes. In the documentation's words, the ability not to label as positive a sample that is negative.
- Recall = tp / (tp + fn): of the cases that were yes, how many you called yes. The ability to find all the positive samples.
A threshold turns each P(yes) into a yes or a no, so every threshold has its own tp, fp and fn, and its own precision and recall.
The ten cases below are illustrative, made up to show the mechanics. Five are really yes, five really no:
- really yes: P(yes) of 0.95, 0.88, 0.81, 0.66, 0.42
- really no: P(yes) of 0.72, 0.55, 0.35, 0.20, 0.08
| Threshold | Called yes | Right yeses (tp) | Wrong yeses (fp) | Precision | Recall |
|---|---|---|---|---|---|
| 0.30 | 8 | 5 | 3 | 0.63 | 1.00 |
| 0.50 | 6 | 4 | 2 | 0.67 | 0.80 |
| 0.70 | 4 | 3 | 1 | 0.75 | 0.60 |
| 0.85 | 2 | 2 | 0 | 1.00 | 0.40 |
Recall can only go down as the threshold rises, since you call fewer cases yes. Precision usually goes up, but not always: on real data it can dip when a threshold drops a right yes before the next wrong one. That is why you read the whole curve, not two points.
Moving a real email to the spam folder can hide an invoice or a job offer; letting one spam message into the inbox costs a second of the reader's attention. The expensive error is the wrong yes, so pick the threshold for precision, and accept that some spam gets through.
Two things help beyond the threshold. Ask several narrow questions ("does it promote a product the reader did not ask about?", "does it contain a link to an unrelated site?") instead of one broad "is this spam?", and combine them with sender facts in code. The details are on spam detection with yes/no questions, and the arithmetic of combining answers is on combining yes/no answers with AND, OR and NOT.
"Does this message ask for a refund?" drives two different actions:
- Routing to the refund queue. Missing a request leaves a customer waiting; a needless routing costs an agent a few seconds. Favour recall with a low threshold.
- Paying out automatically. A wrong yes sends money that was not owed. Favour precision with a high threshold, and send the middle to a person, as on human in the loop AI with a review band.
The README example shows the kind of number involved: for "The box arrived empty. This is the
second time!", refund came back at 0.93. That clears a routing threshold easily, while a payout threshold
set at 0.95 would still hold it for a person, which is the design working as intended.
A measured reason to set the payout bar high: on 999 yes/no questions written after the model was finished, jevos made 152 wrong yeses against 91 wrong noes. On that set its mistakes cost more precision than recall, and a payout decision needs precision most.
Recall is computed only over the real yeses, so it does not change when yes becomes rarer. Precision does: with the same model and threshold, fewer real yeses means the wrong yeses from the large pool of noes make up a bigger share of everything you call yes. Saito and Rehmsmeier (2015) argue for this reason that precision-recall plots are more informative than ROC plots on imbalanced data, because they evaluate the fraction of true positives among positive predictions. The worked arithmetic is on base rates: why a 0.9 yes can still be wrong often.
The practical rule: measure precision on a sample with your real mix of yes and no. A balanced test set gives you a recall you can trust and a precision you cannot.
With the true answers y (1 for yes) and the probabilities p from your labelled requests:
from sklearn.metrics import precision_recall_curve
precision, recall, thresholds = precision_recall_curve(y, p)scikit-learn returns one pair per distinct score; the first pair corresponds to calling everything yes (precision equal to the share of yes, recall 1), and the last precision and recall values are 1 and 0 with no threshold attached. Plot recall on the x-axis and precision on the y-axis, mark the thresholds you are considering, and choose with the action in mind.
What is the difference between precision and recall? Precision is how many of your yeses were right; recall is how many of the real yeses you found.
Does a higher threshold always increase precision? Usually, not always. Recall always goes down or stays the same.
Which should I optimise? The one that protects against the expensive error of the action: precision when a wrong yes is costly, recall when a miss is.
Can one question have two thresholds? Yes. Use a low one for cheap actions such as routing, and a high one for costly actions such as paying out.
Why is my precision lower in production than in testing? Probably because yes is rarer in production than in your test set.
See also: how to choose a threshold for P(yes), evaluation metrics for yes/no classifiers and support ticket routing with yes/no questions.
- Our measurements: 152 wrong yeses vs 91 wrong noes, 999-question set,
jevos-q4_k_m. The refund value 0.93 is the README example of the jev repository. - The ten-case table is illustrative, invented for this page.
- scikit-learn precision_recall_curve, definitions and boundary values, fetched 2026-09-29.
- Saito and Rehmsmeier (2015), The Precision-Recall Plot Is More Informative than the ROC Plot When Evaluating Binary Classifiers on Imbalanced Datasets, PLOS ONE, fetched 2026-09-29.
From the notes of jev. The refund question is the same question at both thresholds; only the action it triggers changes.
- Ask a local LLM a yes/no question and get P(yes)
- Zero-shot text classification with yes/no questions
- LLM policy decisions: put the rule in the question
- LLM as a judge on a CPU
- Why a small LLM says yes when the answer is no
- Small LLMs and arithmetic in yes/no questions
- Our held-out benchmark said 0.855, new questions said 0.757
- jevos vs Jev vs Laya for yes/no decisions
- An open-source alternative to Jev for yes/no decisions
- jevos vs the OpenAI API for yes/no classification
- jevos vs Ollama for yes/no decisions
- jevos vs bart-large-mnli for zero-shot classification
- A yes/no LLM vs a fine-tuned BERT classifier
- jevos vs SetFit: zero-shot vs few-shot classification
- jevos vs Llama Guard for content safety checks
- jev serve vs llama.cpp server for classification
- jevos vs LM Studio: a decision server, not a chat app
- Local vs hosted LLM decisions: latency, cost, privacy
- A yes/no LLM vs a business rules engine
- LLM decisions vs keyword rules and regex
- The fastest AI model for yes/no decisions
- What makes a local LLM fast on a CPU
- Why one forward pass beats generating an answer
- Prefill vs decode: where LLM latency comes from
- Why LLM latency grows with the length of the text
- Why a hosted LLM API cannot answer in 50 ms
- Many questions about one text: why the extra ones are cheap
- CPU or GPU for a small LLM
- Latency budgets: where a 200 ms model fits
- Measuring LLM latency: median, p90 and warm-up
- Q4_K_M vs Q8_0: speed and size for a small model
- Throughput vs latency for a decision server
- What P(yes) means, and what it does not
- LLM calibration explained with yes/no answers
- Expected calibration error (ECE), explained
- Temperature scaling for LLM probabilities
- Platt scaling for a yes/no model
- Reading a reliability diagram
- How to choose a threshold for P(yes)
- Thresholds when a wrong yes costs more than a wrong no
- Human in the loop AI with a review band
- Precision and recall at a P(yes) threshold
- Base rates: why a 0.9 yes can still be wrong often
- Combining yes/no answers with AND, OR and NOT
- Logits, log-odds and P(yes)
- LLM confidence scores: probabilities vs self-reports
- How to write yes/no questions an LLM answers well
- Negation in yes/no questions for an LLM
- One condition per question: splitting compound questions
- Ask whether the text says it at all
- Scores as yes/no thresholds: is it at least high?
- Sending JSON as the text: designing the state
- Why wording changes an LLM's answer, and how to test it
- Mainly about: questions for messages with several topics
- Yes/no questions about tone and emotion
- Asking about intent: what does the writer want?
- Yes/no questions about long documents
- Using an English-only LLM with other languages
- Content moderation with a local LLM
- A Discord moderation bot with a local LLM
- Spam detection with yes/no questions
- Review moderation with a local LLM
- Email triage with a local LLM
- Support ticket routing with yes/no questions
- Urgency detection in customer messages
- Sentiment analysis with yes/no questions
- Intent detection with a local LLM
- Lead qualification with yes/no questions
- Fraud case triage with a local LLM
- Phishing email screening with a local LLM
- Log and alert triage with a local LLM
- Checking text for personal data with yes/no questions
- Prompt injection screening with a small model
- Document classification with a local LLM
- Product categorization with yes/no questions
- Contract clause detection with a local LLM
- Refund request triage with a local LLM
- Detecting cancellation intent in customer messages
- RAG evaluation with yes/no questions
- RAG faithfulness check with a local LLM
- Hallucination detection with a local LLM
- LLM regression tests in CI with yes/no checks
- Rubric design for an LLM judge
- Pairwise comparison with a yes/no judge
- LLM judge bias and how to control it
- Evaluation metrics for yes/no classifiers
- Building a yes/no test set for your own data
- Accuracy by kind of question: why one number hides failures
- Generating test questions with answers computed by code
- Benchmark contamination and truly held-out tests
- An LLM router with yes/no questions
- A model cascade: small model first, large model on doubt
- Semantic routing vs yes/no questions
- Gating AI agent tool calls with yes/no checks
- AI agent guardrails with yes/no questions
- Stop conditions for AI agents
- Logging LLM decisions for audit
- Reducing LLM cost with local yes/no decisions
- Replacing chat LLM calls with yes/no questions
- Structured output vs a probability
- A Python client for local LLM decisions
- Calling a local LLM decision server from JavaScript
- Local LLM yes/no decisions in n8n
- A Slack bot that uses local LLM decisions
- Home Assistant automations with local LLM decisions
- A LangChain tool for local yes/no decisions
- Batch decisions from files with jev decide
- Running LLM yes/no checks in GitHub Actions
- Securing a local LLM server with an API key
- curl examples for a local LLM decision API
- Self-hosted AI for decisions
- A private LLM for text classification
- On-premise LLM for business decisions
- GDPR and automated decision-making with an LLM
- Offline AI for decisions: no network needed
- Edge AI decisions on a CPU
- Run an LLM locally without a GPU
- Small language models explained
- When a small model is enough, and when it is not
- An LLM on a laptop: what it can do in real time
- What is GGUF, for someone deploying a classifier
- GGUF quantization types explained: Q4_K_M, Q8_0 and others
- GGUF vs safetensors
- llama.cpp vs Ollama for a classification service
- llama-cpp-python vs calling llama.cpp through ctypes
- llama.cpp on Windows without compiling
- Running llama.cpp CPU only
- Using llama.cpp prebuilt binaries instead of building