-
Notifications
You must be signed in to change notification settings - Fork 133
zero shot text classification yes no questions
You can classify text into labels with no training data by asking one yes/no question per label and taking the label with the highest P(yes). "Is this a billing problem?", "Is this a shipping problem?", "Is this an account problem?": three questions about the same ticket, sent in one request, give three probabilities, and the classification is the largest of them. On a CPU with jevos, the text is read once for all the questions, so three labels cost much less than three separate calls.
That is zero-shot classification in the literal sense: the labels are written in plain English at request time, and changing them is editing a string, not retraining a model.
This page is the request, the code that turns probabilities into a label, the two cases a naive argmax gets wrong, what to do with ordered levels, and where the approach stops working.
The usual zero-shot setups hand the model the list of labels and ask it to pick one. A chat model does this by generating the label name, which you then have to match against your list; an entailment model does it by scoring "This text is about billing" against the text, one label at a time.
Asking a yes/no question per label is the second idea with a model built for it. Each question is answered on its own, so a label can be described as precisely as you need ("Is the customer asking for money back, not just complaining?") without the other labels getting in the way, and you can add a label without touching the rest. How this compares with the entailment approach is on jevos vs bart-large-mnli, and with a few labelled examples per class on jevos vs SetFit.
The cost is that the probabilities are independent. They do not sum to 1, and two of them can both be high. The code below deals with that explicitly.
{
"model": "jev-latest",
"state": "Hi, my card was charged but the order page still says payment pending, and now I can't log in to check it.",
"questions": {
"billing": {"type": "noul", "instructions": "Is this message mainly about a payment, a charge or a refund?"},
"shipping": {"type": "noul", "instructions": "Is this message mainly about delivery or tracking of a parcel?"},
"account": {"type": "noul", "instructions": "Is this message mainly about logging in or account access?"}
}
}Three habits in those instructions matter more than the model:
- "Mainly". Real messages touch several topics. Asking whether a topic is the main one gives the model a reason to say no to the secondary ones.
- Say what the label covers. "A payment, a charge or a refund" is a better label than "billing", because it is what the model can check against the text.
- Keep them parallel. Questions with the same structure produce probabilities you can compare with each other.
Each question comes back as its own noul, the probability that the answer is yes.
import requests
LABELS = {
"billing": "Is this message mainly about a payment, a charge or a refund?",
"shipping": "Is this message mainly about delivery or tracking of a parcel?",
"account": "Is this message mainly about logging in or account access?",
}
def classify(text, threshold=0.5):
body = {"model": "jev-latest", "state": text,
"questions": {k: {"type": "noul", "instructions": q} for k, q in LABELS.items()}}
answers = requests.post("http://127.0.0.1:8017/v1/systemone", json=body).json()["answers"]
p = {k: a["noul"] for k, a in answers.items()}
best = max(p, key=p.get)
if p[best] < threshold:
return "other", p # no label fits
return best, pmax picks the label; the threshold decides whether any label fits at all.
Nothing fits. If every probability is low, the message belongs to none of your labels, and
the largest of three small numbers is still a small number. Returning "other" below a
threshold, as above, is the fix. A separate "other" label written as a question ("Is this about
none of: payments, deliveries, accounts?") is the weaker design, because a negative definition
gives the model nothing in the text to point at.
Two fit. The example message above really is about a charge and about logging in. Two high probabilities are information, not noise. For routing, pick the larger and log the pair; for tagging, return every label above the threshold instead of only the best one.
For a scale such as low, medium, high, critical, do not ask "Is the priority medium?". Ask one question per boundary:
"questions": {
"at_least_medium": {"type": "noul", "instructions": "Is the urgency of this message at least medium?"},
"at_least_high": {"type": "noul", "instructions": "Is the urgency of this message at least high?"},
"at_least_critical": {"type": "noul", "instructions": "Is the urgency of this message critical?"}
}The level is the highest boundary answered yes. Boundary questions have one clear answer each, where "is it medium?" has two ways to be wrong, and a scale of n levels needs only n minus 1 questions.
On the README's three-question example, three questions about one text take about 66 ms together, against 49 ms for one alone, on an Intel Core Ultra 7 255H with 16 threads. The text is the expensive part and it is read once; each extra label adds a fraction of that. So ten labels on one ticket are one request, not ten. A deep taxonomy asked level by level is on product categorization with yes/no questions.
The limit is context, 8,192 tokens for the text plus each question, counted per question, which leaves room for long documents and many labels before it matters.
On 999 yes/no questions written after training, the question kinds a classifier needs are the model's strongest: 0.954 on facts stated in the text, 0.938 on tone, 0.893 on paraphrases and 0.859 on intent. Topic routing, sentiment, urgency and "does this message ask for X" all live there.
It is weaker when the label depends on a computation: a total over a limit, a date inside a window, a count. Those questions scored 0.58 to 0.65 on the same set, and they are better answered by extracting the number and comparing it in code. The measurement is on small LLMs and arithmetic in yes/no questions.
And it reads English only.
What is zero-shot text classification? Classifying text into labels the model was not trained on, described in words at request time, with no labelled examples.
Can a small local LLM do zero-shot classification? Yes, if each label is asked as its own yes/no question. jevos answers each in one pass on a CPU.
Why do the probabilities not sum to 1? Each question is answered independently. Use a threshold for "no label fits" and allow several labels when more than one is high.
How many labels can I use? Up to 1,024 questions per request, each fitting in 8,192 tokens with the text. They share one reading of the text, so cost grows slowly.
Is it better than fine-tuning a classifier? It is better when labels change or you have no data. With thousands of labelled examples and fixed labels, a fine-tuned classifier will usually win on accuracy.
See also: ask a local LLM a yes/no question, LLM as a judge on a CPU and why a small LLM says yes when the answer is no.
- Timing for several questions about one text, and the 8,192-token context: the jev README.
- Accuracy by question kind: our 999-question test set, written after training and never used
for tuning, run on
jevos-q4_k_m.
From the notes of jev, a yes/no decision model that runs on a laptop CPU. Of the code on this page, the threshold for "other" is the line that matters most.
- Ask a local LLM a yes/no question and get P(yes)
- Zero-shot text classification with yes/no questions
- LLM policy decisions: put the rule in the question
- LLM as a judge on a CPU
- Why a small LLM says yes when the answer is no
- Small LLMs and arithmetic in yes/no questions
- Our held-out benchmark said 0.855, new questions said 0.757
- jevos vs Jev vs Laya for yes/no decisions
- An open-source alternative to Jev for yes/no decisions
- jevos vs the OpenAI API for yes/no classification
- jevos vs Ollama for yes/no decisions
- jevos vs bart-large-mnli for zero-shot classification
- A yes/no LLM vs a fine-tuned BERT classifier
- jevos vs SetFit: zero-shot vs few-shot classification
- jevos vs Llama Guard for content safety checks
- jev serve vs llama.cpp server for classification
- jevos vs LM Studio: a decision server, not a chat app
- Local vs hosted LLM decisions: latency, cost, privacy
- A yes/no LLM vs a business rules engine
- LLM decisions vs keyword rules and regex
- The fastest AI model for yes/no decisions
- What makes a local LLM fast on a CPU
- Why one forward pass beats generating an answer
- Prefill vs decode: where LLM latency comes from
- Why LLM latency grows with the length of the text
- Why a hosted LLM API cannot answer in 50 ms
- Many questions about one text: why the extra ones are cheap
- CPU or GPU for a small LLM
- Latency budgets: where a 200 ms model fits
- Measuring LLM latency: median, p90 and warm-up
- Q4_K_M vs Q8_0: speed and size for a small model
- Throughput vs latency for a decision server
- What P(yes) means, and what it does not
- LLM calibration explained with yes/no answers
- Expected calibration error (ECE), explained
- Temperature scaling for LLM probabilities
- Platt scaling for a yes/no model
- Reading a reliability diagram
- How to choose a threshold for P(yes)
- Thresholds when a wrong yes costs more than a wrong no
- Human in the loop AI with a review band
- Precision and recall at a P(yes) threshold
- Base rates: why a 0.9 yes can still be wrong often
- Combining yes/no answers with AND, OR and NOT
- Logits, log-odds and P(yes)
- LLM confidence scores: probabilities vs self-reports
- How to write yes/no questions an LLM answers well
- Negation in yes/no questions for an LLM
- One condition per question: splitting compound questions
- Ask whether the text says it at all
- Scores as yes/no thresholds: is it at least high?
- Sending JSON as the text: designing the state
- Why wording changes an LLM's answer, and how to test it
- Mainly about: questions for messages with several topics
- Yes/no questions about tone and emotion
- Asking about intent: what does the writer want?
- Yes/no questions about long documents
- Using an English-only LLM with other languages
- Content moderation with a local LLM
- A Discord moderation bot with a local LLM
- Spam detection with yes/no questions
- Review moderation with a local LLM
- Email triage with a local LLM
- Support ticket routing with yes/no questions
- Urgency detection in customer messages
- Sentiment analysis with yes/no questions
- Intent detection with a local LLM
- Lead qualification with yes/no questions
- Fraud case triage with a local LLM
- Phishing email screening with a local LLM
- Log and alert triage with a local LLM
- Checking text for personal data with yes/no questions
- Prompt injection screening with a small model
- Document classification with a local LLM
- Product categorization with yes/no questions
- Contract clause detection with a local LLM
- Refund request triage with a local LLM
- Detecting cancellation intent in customer messages
- RAG evaluation with yes/no questions
- RAG faithfulness check with a local LLM
- Hallucination detection with a local LLM
- LLM regression tests in CI with yes/no checks
- Rubric design for an LLM judge
- Pairwise comparison with a yes/no judge
- LLM judge bias and how to control it
- Evaluation metrics for yes/no classifiers
- Building a yes/no test set for your own data
- Accuracy by kind of question: why one number hides failures
- Generating test questions with answers computed by code
- Benchmark contamination and truly held-out tests
- An LLM router with yes/no questions
- A model cascade: small model first, large model on doubt
- Semantic routing vs yes/no questions
- Gating AI agent tool calls with yes/no checks
- AI agent guardrails with yes/no questions
- Stop conditions for AI agents
- Logging LLM decisions for audit
- Reducing LLM cost with local yes/no decisions
- Replacing chat LLM calls with yes/no questions
- Structured output vs a probability
- A Python client for local LLM decisions
- Calling a local LLM decision server from JavaScript
- Local LLM yes/no decisions in n8n
- A Slack bot that uses local LLM decisions
- Home Assistant automations with local LLM decisions
- A LangChain tool for local yes/no decisions
- Batch decisions from files with jev decide
- Running LLM yes/no checks in GitHub Actions
- Securing a local LLM server with an API key
- curl examples for a local LLM decision API
- Self-hosted AI for decisions
- A private LLM for text classification
- On-premise LLM for business decisions
- GDPR and automated decision-making with an LLM
- Offline AI for decisions: no network needed
- Edge AI decisions on a CPU
- Run an LLM locally without a GPU
- Small language models explained
- When a small model is enough, and when it is not
- An LLM on a laptop: what it can do in real time
- What is GGUF, for someone deploying a classifier
- GGUF quantization types explained: Q4_K_M, Q8_0 and others
- GGUF vs safetensors
- llama.cpp vs Ollama for a classification service
- llama-cpp-python vs calling llama.cpp through ctypes
- llama.cpp on Windows without compiling
- Running llama.cpp CPU only
- Using llama.cpp prebuilt binaries instead of building