-
Notifications
You must be signed in to change notification settings - Fork 132
what p yes means
P(yes) is the model's probability that the answer to your question, about this text, is yes. From a calibrated model, 0.9 means that among many answers of about 0.9 on similar data, about nine in ten turn out to be yes. It is not a certainty, it is not a measure of how much of something the text contains, and it says nothing checkable about the one case in front of you on its own. Its meaning lives in the long run, over many answers, and only on data like the data it was checked on.
That last condition is the one people skip. A probability is a promise about frequencies, and the promise is only as good as the match between your inputs and the inputs it was measured on. When the inputs change, the number keeps looking the same while the promise quietly breaks.
This page is the three readings of a number like 0.9, what it promises over many answers, what it cannot tell you about one, when it stops meaning what it says, and how to use it in code.
The same number can be read three ways, and only one of them is right for a calibrated model.
- A frequency over similar cases. Of the texts and questions that get about 0.9, about 90 percent are yes. This is the reading calibration is about, and the one scikit-learn's documentation uses: a well calibrated classifier is one where, among the samples given a value close to 0.8, about 80 percent belong to the positive class.
- A certainty. "The model is 90 percent sure." There is no inner feeling being reported. It is a number produced by the model for this input, and the only way to know what it is worth is to count how often such numbers were right.
-
A score or intensity. "The customer is 0.73 upset." This reading is wrong. In the README
example,
upsetcame back at 0.73 for "The box arrived empty. This is the second time!". That is the probability that the answer to "Is the customer upset?" is yes, not a measure of how angry the customer is. A mildly annoyed customer and a furious one can both get a high P(yes); to ask about degree, ask a threshold question such as "is the customer very upset?", covered in scores as yes/no thresholds.
A score that is not calibrated can still be useful for ranking: a higher number means more likely yes. But you cannot read it as "nine in ten" until you have checked it.
On data like the data it was checked on, a calibrated P(yes) lets you plan. If you act on every answer above 0.9, you can expect roughly one wrong action in ten or fewer among them; if you average the probabilities over a batch, you get a fair estimate of how many are yes.
For jevos, the check we have is this: on 6,397 natural yes/no questions from a held-out split,
the calibration error (ECE) of jevos-q8_0 was 0.009. In plain terms, on that data, the numbers
the model gave were very close to the frequencies it achieved. What ECE measures, and how, is on
expected calibration error, explained.
A single answer of 0.9 does not come with a 90 percent that you can verify. The case is either yes or no. The 0.9 tells you which bucket of cases it belongs to, and the bucket has a track record.
Two practical consequences:
- One wrong 0.95 is not evidence of a broken model. At that level you should expect about one in twenty to be wrong. Ten wrong 0.95s in a row is evidence.
- The number does not explain itself. The README refund example returned 0.78 for "Should this customer get a refund?" That is not "78 percent of a refund", and it does not say which part of the policy the model weighed. If you need to know why, split the question into its conditions and ask each one; see combining yes/no answers.
Calibration is a property of a model on a kind of data, not of the model alone. On 999 yes/no
questions written from scratch after the model was finished, of new kinds and in new scenarios,
jevos-q4_k_m was right 0.757 of the time, and its probabilities leaned toward yes: on
arithmetic questions whose true answer was no, the average P(yes) was 0.59. A calibrated model
would put those near 0. The same model that looks calibrated on familiar data is not calibrated
on questions it cannot work out. The detail is on
why a small LLM says yes when the answer is no, and the gap
between the two test sets on
our held-out benchmark said 0.855.
The base rate matters too. If yes is rare in your traffic, a model calibrated on a more balanced mix will give yeses that are wrong more often than the number suggests; base rates works through why.
Keep the number, threshold it, and log it. The README pattern is the whole idea:
if answer["answers"]["billing"]["noul"] > 0.5:
print("send to billing")From there, three habits pay off. Choose the threshold from your own labelled cases, not by taste (how to choose a threshold). Send the middle of the range to a person when a mistake is expensive (a review band). And store the probability next to the decision, so you can later check whether your 0.9s were right nine times in ten.
Does P(yes) = 0.9 mean the model is 90 percent sure? It means that, on similar data, about nine in ten answers of that size were yes. It is a frequency you can check, not a feeling.
Is a higher P(yes) a stronger yes? It is a more likely yes, not a bigger one. To ask about degree, ask a separate threshold question.
Can I compare P(yes) across different questions? Only if each question is calibrated on your data. Some kinds of question lean toward yes, so the same 0.7 can mean different things.
Why did a 0.95 answer turn out wrong? At 0.95, about one in twenty should be. Judge the number over many answers, not one.
Is P(yes) the same as accuracy? No. Accuracy is how often the thresholded answer is right; P(yes) is the model's estimate for one case.
See also: LLM calibration explained, reading a reliability diagram and LLM confidence scores: probabilities vs self-reports.
- Our measurements: ECE 0.009 on 6,397 natural yes/no held-out questions (
jevos-q8_0); 0.757 accuracy and mean P(yes) 0.59 on no-answer arithmetic questions from the 999-question set (jevos-q4_k_m). - The billing, refund and upset values are the README examples of the jev repository.
- Definition of a well calibrated classifier: scikit-learn, Probability calibration, fetched 2026-09-29.
From the notes of jev, which returns one number per yes/no question and nothing else, so this page is about the only output there is.
- Ask a local LLM a yes/no question and get P(yes)
- Zero-shot text classification with yes/no questions
- LLM policy decisions: put the rule in the question
- LLM as a judge on a CPU
- Why a small LLM says yes when the answer is no
- Small LLMs and arithmetic in yes/no questions
- Our held-out benchmark said 0.855, new questions said 0.757
- jevos vs Jev vs Laya for yes/no decisions
- An open-source alternative to Jev for yes/no decisions
- jevos vs the OpenAI API for yes/no classification
- jevos vs Ollama for yes/no decisions
- jevos vs bart-large-mnli for zero-shot classification
- A yes/no LLM vs a fine-tuned BERT classifier
- jevos vs SetFit: zero-shot vs few-shot classification
- jevos vs Llama Guard for content safety checks
- jev serve vs llama.cpp server for classification
- jevos vs LM Studio: a decision server, not a chat app
- Local vs hosted LLM decisions: latency, cost, privacy
- A yes/no LLM vs a business rules engine
- LLM decisions vs keyword rules and regex
- The fastest AI model for yes/no decisions
- What makes a local LLM fast on a CPU
- Why one forward pass beats generating an answer
- Prefill vs decode: where LLM latency comes from
- Why LLM latency grows with the length of the text
- Why a hosted LLM API cannot answer in 50 ms
- Many questions about one text: why the extra ones are cheap
- CPU or GPU for a small LLM
- Latency budgets: where a 200 ms model fits
- Measuring LLM latency: median, p90 and warm-up
- Q4_K_M vs Q8_0: speed and size for a small model
- Throughput vs latency for a decision server
- What P(yes) means, and what it does not
- LLM calibration explained with yes/no answers
- Expected calibration error (ECE), explained
- Temperature scaling for LLM probabilities
- Platt scaling for a yes/no model
- Reading a reliability diagram
- How to choose a threshold for P(yes)
- Thresholds when a wrong yes costs more than a wrong no
- Human in the loop AI with a review band
- Precision and recall at a P(yes) threshold
- Base rates: why a 0.9 yes can still be wrong often
- Combining yes/no answers with AND, OR and NOT
- Logits, log-odds and P(yes)
- LLM confidence scores: probabilities vs self-reports
- How to write yes/no questions an LLM answers well
- Negation in yes/no questions for an LLM
- One condition per question: splitting compound questions
- Ask whether the text says it at all
- Scores as yes/no thresholds: is it at least high?
- Sending JSON as the text: designing the state
- Why wording changes an LLM's answer, and how to test it
- Mainly about: questions for messages with several topics
- Yes/no questions about tone and emotion
- Asking about intent: what does the writer want?
- Yes/no questions about long documents
- Using an English-only LLM with other languages
- Content moderation with a local LLM
- A Discord moderation bot with a local LLM
- Spam detection with yes/no questions
- Review moderation with a local LLM
- Email triage with a local LLM
- Support ticket routing with yes/no questions
- Urgency detection in customer messages
- Sentiment analysis with yes/no questions
- Intent detection with a local LLM
- Lead qualification with yes/no questions
- Fraud case triage with a local LLM
- Phishing email screening with a local LLM
- Log and alert triage with a local LLM
- Checking text for personal data with yes/no questions
- Prompt injection screening with a small model
- Document classification with a local LLM
- Product categorization with yes/no questions
- Contract clause detection with a local LLM
- Refund request triage with a local LLM
- Detecting cancellation intent in customer messages
- RAG evaluation with yes/no questions
- RAG faithfulness check with a local LLM
- Hallucination detection with a local LLM
- LLM regression tests in CI with yes/no checks
- Rubric design for an LLM judge
- Pairwise comparison with a yes/no judge
- LLM judge bias and how to control it
- Evaluation metrics for yes/no classifiers
- Building a yes/no test set for your own data
- Accuracy by kind of question: why one number hides failures
- Generating test questions with answers computed by code
- Benchmark contamination and truly held-out tests
- An LLM router with yes/no questions
- A model cascade: small model first, large model on doubt
- Semantic routing vs yes/no questions
- Gating AI agent tool calls with yes/no checks
- AI agent guardrails with yes/no questions
- Stop conditions for AI agents
- Logging LLM decisions for audit
- Reducing LLM cost with local yes/no decisions
- Replacing chat LLM calls with yes/no questions
- Structured output vs a probability
- A Python client for local LLM decisions
- Calling a local LLM decision server from JavaScript
- Local LLM yes/no decisions in n8n
- A Slack bot that uses local LLM decisions
- Home Assistant automations with local LLM decisions
- A LangChain tool for local yes/no decisions
- Batch decisions from files with jev decide
- Running LLM yes/no checks in GitHub Actions
- Securing a local LLM server with an API key
- curl examples for a local LLM decision API
- Self-hosted AI for decisions
- A private LLM for text classification
- On-premise LLM for business decisions
- GDPR and automated decision-making with an LLM
- Offline AI for decisions: no network needed
- Edge AI decisions on a CPU
- Run an LLM locally without a GPU
- Small language models explained
- When a small model is enough, and when it is not
- An LLM on a laptop: what it can do in real time
- What is GGUF, for someone deploying a classifier
- GGUF quantization types explained: Q4_K_M, Q8_0 and others
- GGUF vs safetensors
- llama.cpp vs Ollama for a classification service
- llama-cpp-python vs calling llama.cpp through ctypes
- llama.cpp on Windows without compiling
- Running llama.cpp CPU only
- Using llama.cpp prebuilt binaries instead of building