-
Notifications
You must be signed in to change notification settings - Fork 132
yes no questions about intent
Ask about intent with a verb of wanting and a named writer ("Does the customer ask for a refund?", "Does the customer say they want to cancel?"), and decide up front whether you mean what the writer states or what they only imply. Stated intent is a reading question and one jevos handles well: on the intent questions of our 999-question test set it was right 0.859 of the time. Implied intent is a judgment, useful and less reliable, and it deserves its own question and its own threshold rather than being folded into the stated one.
Intent is usually the question a support system actually needs answered. Topic says which team; tone says how the writer feels; intent says what should happen next. A message about billing that asks for nothing needs a different action from one that asks for a refund, and a calm message that says "please close my account" matters more than an angry one that does not.
This page is the three intents most systems care about, how to phrase each, the line between stated and implied, and what the measurement does and does not cover.
Most customer messages carry one of a few intents, and they are worth separate questions because they lead to separate actions:
{
"model": "jev-latest",
"state": "This is the second time my order arrived damaged. I want a replacement sent this week, otherwise I will cancel my subscription.",
"questions": {
"asks_replacement": {"type": "noul", "instructions": "Does the customer ask for a replacement?"},
"asks_refund": {"type": "noul", "instructions": "Does the customer ask for a refund?"},
"complains": {"type": "noul", "instructions": "Is the customer complaining about a product or service?"},
"cancel_threat": {"type": "noul", "instructions": "Does the customer say they will cancel if the problem is not fixed?"}
}
}- A request names something the writer wants done. Phrase it with "ask for", "request" or "want", followed by the thing: a replacement, a refund, a callback, a change of address.
- A complaint reports that something went wrong, with or without a request. Many complaints ask for nothing, and treating them as requests creates work nobody asked for.
- A threat to leave is conditional: "otherwise I will cancel". It is a different question from "Does the customer ask to cancel?", which is an unconditional request, and the two lead to different teams. The detection side is on detecting cancellation intent in customer messages.
Asking "refund" and "replacement" as separate questions matters even when both go to the same team. The customer above asked for a replacement, not money, and a reply that offers a refund answers the wrong request.
"I want a refund" states the intent. "I have had this for two days and it already stopped working" implies it: a reader guesses the writer would like a refund or a replacement, but the writer has not said so.
Both are real, and they are different questions:
| Question | Asks about | Typical use |
|---|---|---|
| Does the customer ask for a refund? | stated intent | automatic actions, templates |
| Does the customer seem to expect a refund, even if they do not ask for one? | implied intent | prioritising, suggesting a reply |
The first question has an answer in the words on the page. The second asks the model to infer, and people disagree on inferences too. Keep them apart so that an automatic action never runs on an inference. The same principle, asking whether the text says a thing at all, is on ask whether the text says it at all.
Verbs set the bar. "Mention a refund" is weaker than "ask for a refund", which is weaker than "demand a refund". Pick the verb that matches the action you will take, and test the choice as on why wording changes the answer.
On the 999 yes/no questions written after the model was finished (10 scenarios, 10 texts each,
emails, tickets, logs, reviews and forms, exactly half of the answers yes), 71 were intent
questions. jevos q4_k_m answered 0.859 of them correctly, and its average P(yes) on intent
questions whose correct answer was no was 0.28.
For comparison on the same set: stated facts 0.954, tone 0.938, negation 0.858. Intent sits with the reading kinds, below the plainest of them. Two things the set does not tell you: how the 0.859 splits between stated and implied intent, since we did not label them separately, and how it holds on text very different from business writing.
The 0.28 is worth a second look when a wrong yes is expensive. It is about the same as on stated facts (0.29) and far from the 0.59 on arithmetic, so intent questions do not carry a strong lean toward yes, but for an automatic refund a threshold above 0.5 is still sensible. How to pick it is on thresholds when a wrong yes costs more.
Intent works best as one of three questions about a message, each answering a different part of the decision:
- topic picks the team ("Is this mainly about a payment?"),
- intent picks the action ("Does the customer ask for a refund?"),
- tone picks the urgency and the care ("Is the customer angry?").
They share one reading of the text, so asking all three costs little more than asking one. Tone has its own page, yes/no questions about tone and emotion, and the routing that ties the three together is on intent detection with a local LLM.
How do I detect what a customer wants? Ask one yes/no question per intent, with a verb of wanting and the thing wanted: "Does the customer ask for a replacement?".
How accurate is a small LLM on intent? On our 999-question test set jevos answered 0.859 of 71 intent questions correctly.
Is a complaint the same as a request? No. Many complaints ask for nothing. Ask both, and act on the request.
Can the model detect implied intent? You can ask it, as a separate question. Treat the answer as a judgment and do not run automatic actions on it.
What is the difference between a cancellation request and a threat to cancel? The request is unconditional; the threat depends on something else happening. Ask them as two questions.
See also: refund request triage with a local LLM, support ticket routing with yes/no questions and how to write yes/no questions an LLM answers well.
- 0.859 on 71 intent questions, the other accuracies by kind, and the mean P(yes) values on
no-answer questions: our 999-question test set,
jevos-q4_k_m.
From the notes of jev, where the difference between "asks
for" and "mentions" is one word in instructions and a different action downstream.
- Ask a local LLM a yes/no question and get P(yes)
- Zero-shot text classification with yes/no questions
- LLM policy decisions: put the rule in the question
- LLM as a judge on a CPU
- Why a small LLM says yes when the answer is no
- Small LLMs and arithmetic in yes/no questions
- Our held-out benchmark said 0.855, new questions said 0.757
- jevos vs Jev vs Laya for yes/no decisions
- An open-source alternative to Jev for yes/no decisions
- jevos vs the OpenAI API for yes/no classification
- jevos vs Ollama for yes/no decisions
- jevos vs bart-large-mnli for zero-shot classification
- A yes/no LLM vs a fine-tuned BERT classifier
- jevos vs SetFit: zero-shot vs few-shot classification
- jevos vs Llama Guard for content safety checks
- jev serve vs llama.cpp server for classification
- jevos vs LM Studio: a decision server, not a chat app
- Local vs hosted LLM decisions: latency, cost, privacy
- A yes/no LLM vs a business rules engine
- LLM decisions vs keyword rules and regex
- The fastest AI model for yes/no decisions
- What makes a local LLM fast on a CPU
- Why one forward pass beats generating an answer
- Prefill vs decode: where LLM latency comes from
- Why LLM latency grows with the length of the text
- Why a hosted LLM API cannot answer in 50 ms
- Many questions about one text: why the extra ones are cheap
- CPU or GPU for a small LLM
- Latency budgets: where a 200 ms model fits
- Measuring LLM latency: median, p90 and warm-up
- Q4_K_M vs Q8_0: speed and size for a small model
- Throughput vs latency for a decision server
- What P(yes) means, and what it does not
- LLM calibration explained with yes/no answers
- Expected calibration error (ECE), explained
- Temperature scaling for LLM probabilities
- Platt scaling for a yes/no model
- Reading a reliability diagram
- How to choose a threshold for P(yes)
- Thresholds when a wrong yes costs more than a wrong no
- Human in the loop AI with a review band
- Precision and recall at a P(yes) threshold
- Base rates: why a 0.9 yes can still be wrong often
- Combining yes/no answers with AND, OR and NOT
- Logits, log-odds and P(yes)
- LLM confidence scores: probabilities vs self-reports
- How to write yes/no questions an LLM answers well
- Negation in yes/no questions for an LLM
- One condition per question: splitting compound questions
- Ask whether the text says it at all
- Scores as yes/no thresholds: is it at least high?
- Sending JSON as the text: designing the state
- Why wording changes an LLM's answer, and how to test it
- Mainly about: questions for messages with several topics
- Yes/no questions about tone and emotion
- Asking about intent: what does the writer want?
- Yes/no questions about long documents
- Using an English-only LLM with other languages
- Content moderation with a local LLM
- A Discord moderation bot with a local LLM
- Spam detection with yes/no questions
- Review moderation with a local LLM
- Email triage with a local LLM
- Support ticket routing with yes/no questions
- Urgency detection in customer messages
- Sentiment analysis with yes/no questions
- Intent detection with a local LLM
- Lead qualification with yes/no questions
- Fraud case triage with a local LLM
- Phishing email screening with a local LLM
- Log and alert triage with a local LLM
- Checking text for personal data with yes/no questions
- Prompt injection screening with a small model
- Document classification with a local LLM
- Product categorization with yes/no questions
- Contract clause detection with a local LLM
- Refund request triage with a local LLM
- Detecting cancellation intent in customer messages
- RAG evaluation with yes/no questions
- RAG faithfulness check with a local LLM
- Hallucination detection with a local LLM
- LLM regression tests in CI with yes/no checks
- Rubric design for an LLM judge
- Pairwise comparison with a yes/no judge
- LLM judge bias and how to control it
- Evaluation metrics for yes/no classifiers
- Building a yes/no test set for your own data
- Accuracy by kind of question: why one number hides failures
- Generating test questions with answers computed by code
- Benchmark contamination and truly held-out tests
- An LLM router with yes/no questions
- A model cascade: small model first, large model on doubt
- Semantic routing vs yes/no questions
- Gating AI agent tool calls with yes/no checks
- AI agent guardrails with yes/no questions
- Stop conditions for AI agents
- Logging LLM decisions for audit
- Reducing LLM cost with local yes/no decisions
- Replacing chat LLM calls with yes/no questions
- Structured output vs a probability
- A Python client for local LLM decisions
- Calling a local LLM decision server from JavaScript
- Local LLM yes/no decisions in n8n
- A Slack bot that uses local LLM decisions
- Home Assistant automations with local LLM decisions
- A LangChain tool for local yes/no decisions
- Batch decisions from files with jev decide
- Running LLM yes/no checks in GitHub Actions
- Securing a local LLM server with an API key
- curl examples for a local LLM decision API
- Self-hosted AI for decisions
- A private LLM for text classification
- On-premise LLM for business decisions
- GDPR and automated decision-making with an LLM
- Offline AI for decisions: no network needed
- Edge AI decisions on a CPU
- Run an LLM locally without a GPU
- Small language models explained
- When a small model is enough, and when it is not
- An LLM on a laptop: what it can do in real time
- What is GGUF, for someone deploying a classifier
- GGUF quantization types explained: Q4_K_M, Q8_0 and others
- GGUF vs safetensors
- llama.cpp vs Ollama for a classification service
- llama-cpp-python vs calling llama.cpp through ctypes
- llama.cpp on Windows without compiling
- Running llama.cpp CPU only
- Using llama.cpp prebuilt binaries instead of building