-
Notifications
You must be signed in to change notification settings - Fork 129
cancellation intent detection
Cancellation intent is three different things that sound alike: a customer asking to cancel now, a customer threatening to cancel if something is not fixed, and a customer venting with no plan to leave. Ask a small local model a separate yes/no question for each, plus a tone question, and route them differently: process the request, send the threat to someone who can fix the problem, and answer the venting as ordinary support. A single "does this customer want to cancel?" question blurs the three, and the cost of the blur runs in both directions.
The direction that is easy to miss is the first one. Treating a clear cancellation request as a retention opportunity, and making the customer ask again, costs trust that no retention offer wins back. A detector that is good at spotting clear requests is as much about processing them promptly as about saving accounts.
This page is how the three kinds differ, the questions that separate them, why intent and tone are separate questions, the routing table, stated and implied intent, and how to check it on your own messages.
| Kind | Example | What the writer wants |
|---|---|---|
| request | "Please cancel my subscription at the end of this month." | the cancellation |
| conditional threat | "If this isn't fixed by Friday I'm cancelling." | the problem fixed |
| venting | "Honestly, this is the worst app I have ever paid for." | to be heard, maybe a fix |
| question about cancelling | "What happens to my data if I cancel?" | information |
The last row matters more than it looks. A customer who asks how cancellation works may be about to leave, or may be checking before renewing. It is a question to answer clearly, not a signal to push back on.
One question per kind, each about what the writer says or asks for:
{
"model": "jev-latest",
"state": {
"plan": "annual",
"message": "Third outage this month. If support can't tell me what happened, I'm moving to another provider."
},
"questions": {
"asks_to_cancel": {"type": "noul", "instructions": "Does the customer ask for their account or subscription to be cancelled?"},
"conditional": {"type": "noul", "instructions": "Does the customer say they will leave only if something is not fixed or explained?"},
"asks_about_exit": {"type": "noul", "instructions": "Does the customer ask how cancelling or leaving works?"},
"upset": {"type": "noul", "instructions": "Is the customer upset?"},
"states_problem": {"type": "noul", "instructions": "Does the customer describe a specific problem with the service?"}
}
}Each question comes back as its own noul, all from one read of the message. "Asks for their
account to be cancelled" is a request; "will leave only if something is not fixed" is the
condition that makes a threat. Asking both, rather than one broad cancellation question, is
what lets the routing tell them apart. The wording of intent questions in general is on
asking about intent: what does the writer want?.
Because they vary independently. A polite "please cancel my account, thanks for everything" is a clear request with no anger. A furious message about an outage may contain no intent to leave at all. If one question mixes them, an angry venting message scores like a request and a calm request scores like nothing.
The two kinds of question also behave differently on our measurements. On 999 yes/no questions written after training, tone questions were answered right 0.938 of the time and intent questions 0.859. On questions whose right answer was no, the mean P(yes) was 0.16 for tone and 0.28 for intent: the model is steadier on how a message feels than on what the writer wants. That is a reason to set a firmer threshold on the intent questions, and to let tone adjust priority rather than route. More on the tone side on yes/no questions about tone and emotion, including the caveats about sarcasm.
| Answers | Route |
|---|---|
asks_to_cancel high |
cancellation processing, confirm promptly; offer alternatives once, if at all |
conditional high |
retention or a senior agent, with the stated problem attached |
asks_about_exit high, others low |
answer the question, with a link to the cancellation page |
upset high, no intent |
ordinary support queue, higher priority |
| everything in the middle | a person reads it |
Two notes on the table. First, a request that is also angry is still a request: process it, and
let tone decide how the reply is written, not whether the customer gets what they asked for.
Second, the conditional threat is the message where a team can change the outcome, because the
customer has said what would keep them. That is where the stated problem, from states_problem
and the message itself, is worth putting in front of someone who can act. Routing mechanics,
queues and fallbacks are on
support ticket routing with yes/no questions.
"I'm cancelling" is stated. "I've started looking at other providers" implies an intent without saying it, and "I just want to know what my options are" may imply one or may not. We have not measured the two separately, but the reason to expect a gap is simple: a stated intent is close to a stated fact, the strongest kind of question on our test set at 0.954, while an implied one has to be inferred.
So write the routing questions about what the customer says ("Does the customer ask...", "Does the customer say they will..."), and treat implied intent as a softer signal: something to flag for an account manager to look at, not something to act on automatically. The difference between a message's several topics and its main one is on mainly about: questions for messages with several topics.
Pull a few hundred past messages that people have already handled, and label each with the four kinds in the first table (several labels are allowed). Run the questions and look at two confusions in particular: requests routed as threats, which delay a customer who has already decided, and threats routed as venting, which lose the chance to fix the problem. Adjust wording first and thresholds second. We have not measured jevos on a cancellation dataset, so this check is the only accuracy figure that applies to your messages. jevos reads English only.
How do I detect churn intent in customer messages? Ask separate yes/no questions for a cancellation request, a conditional threat and a question about leaving, plus a tone question, and route on the combination.
What is the difference between a threat to cancel and a request? A request asks for the cancellation. A threat names a condition, something that must be fixed, and is the one where retention work can help.
Should a retention team see every cancellation request? Not as a hurdle. Process clear requests promptly; send conditional threats to the people who can fix the stated problem.
Is an angry message a churn signal? Not by itself. Tone and intent are separate questions, and on our test set tone was answered more reliably (0.938) than intent (0.859).
Can this run on every incoming message? A handful of questions on one short message takes a fraction of a second on a laptop CPU, and the text stays on your machine.
See also: intent detection with a local LLM, sentiment analysis with yes/no questions and refund request triage with a local LLM.
- Accuracy by kind of question (tone 0.938, intent 0.859, fact 0.954) and the mean P(yes) on
no-answer questions (tone 0.16, intent 0.28): our 999-question test set on
jevos-q4_k_m. No cancellation dataset was measured. - Request format and the one-read-per-request behaviour: the jev README.
- No external facts are stated on this page.
From the notes of jev, a yes/no decision model that runs on a laptop CPU. The example message is a conditional threat on purpose: it contains the words of a cancellation and is not a request, which is the whole reason to ask more than one question.
- Ask a local LLM a yes/no question and get P(yes)
- Zero-shot text classification with yes/no questions
- LLM policy decisions: put the rule in the question
- LLM as a judge on a CPU
- Why a small LLM says yes when the answer is no
- Small LLMs and arithmetic in yes/no questions
- Our held-out benchmark said 0.855, new questions said 0.757
- jevos vs Jev vs Laya for yes/no decisions
- An open-source alternative to Jev for yes/no decisions
- jevos vs the OpenAI API for yes/no classification
- jevos vs Ollama for yes/no decisions
- jevos vs bart-large-mnli for zero-shot classification
- A yes/no LLM vs a fine-tuned BERT classifier
- jevos vs SetFit: zero-shot vs few-shot classification
- jevos vs Llama Guard for content safety checks
- jev serve vs llama.cpp server for classification
- jevos vs LM Studio: a decision server, not a chat app
- Local vs hosted LLM decisions: latency, cost, privacy
- A yes/no LLM vs a business rules engine
- LLM decisions vs keyword rules and regex
- The fastest AI model for yes/no decisions
- What makes a local LLM fast on a CPU
- Why one forward pass beats generating an answer
- Prefill vs decode: where LLM latency comes from
- Why LLM latency grows with the length of the text
- Why a hosted LLM API cannot answer in 50 ms
- Many questions about one text: why the extra ones are cheap
- CPU or GPU for a small LLM
- Latency budgets: where a 200 ms model fits
- Measuring LLM latency: median, p90 and warm-up
- Q4_K_M vs Q8_0: speed and size for a small model
- Throughput vs latency for a decision server
- What P(yes) means, and what it does not
- LLM calibration explained with yes/no answers
- Expected calibration error (ECE), explained
- Temperature scaling for LLM probabilities
- Platt scaling for a yes/no model
- Reading a reliability diagram
- How to choose a threshold for P(yes)
- Thresholds when a wrong yes costs more than a wrong no
- Human in the loop AI with a review band
- Precision and recall at a P(yes) threshold
- Base rates: why a 0.9 yes can still be wrong often
- Combining yes/no answers with AND, OR and NOT
- Logits, log-odds and P(yes)
- LLM confidence scores: probabilities vs self-reports
- How to write yes/no questions an LLM answers well
- Negation in yes/no questions for an LLM
- One condition per question: splitting compound questions
- Ask whether the text says it at all
- Scores as yes/no thresholds: is it at least high?
- Sending JSON as the text: designing the state
- Why wording changes an LLM's answer, and how to test it
- Mainly about: questions for messages with several topics
- Yes/no questions about tone and emotion
- Asking about intent: what does the writer want?
- Yes/no questions about long documents
- Using an English-only LLM with other languages
- Content moderation with a local LLM
- A Discord moderation bot with a local LLM
- Spam detection with yes/no questions
- Review moderation with a local LLM
- Email triage with a local LLM
- Support ticket routing with yes/no questions
- Urgency detection in customer messages
- Sentiment analysis with yes/no questions
- Intent detection with a local LLM
- Lead qualification with yes/no questions
- Fraud case triage with a local LLM
- Phishing email screening with a local LLM
- Log and alert triage with a local LLM
- Checking text for personal data with yes/no questions
- Prompt injection screening with a small model
- Document classification with a local LLM
- Product categorization with yes/no questions
- Contract clause detection with a local LLM
- Refund request triage with a local LLM
- Detecting cancellation intent in customer messages
- RAG evaluation with yes/no questions
- RAG faithfulness check with a local LLM
- Hallucination detection with a local LLM
- LLM regression tests in CI with yes/no checks
- Rubric design for an LLM judge
- Pairwise comparison with a yes/no judge
- LLM judge bias and how to control it
- Evaluation metrics for yes/no classifiers
- Building a yes/no test set for your own data
- Accuracy by kind of question: why one number hides failures
- Generating test questions with answers computed by code
- Benchmark contamination and truly held-out tests
- An LLM router with yes/no questions
- A model cascade: small model first, large model on doubt
- Semantic routing vs yes/no questions
- Gating AI agent tool calls with yes/no checks
- AI agent guardrails with yes/no questions
- Stop conditions for AI agents
- Logging LLM decisions for audit
- Reducing LLM cost with local yes/no decisions
- Replacing chat LLM calls with yes/no questions
- Structured output vs a probability
- A Python client for local LLM decisions
- Calling a local LLM decision server from JavaScript
- Local LLM yes/no decisions in n8n
- A Slack bot that uses local LLM decisions
- Home Assistant automations with local LLM decisions
- A LangChain tool for local yes/no decisions
- Batch decisions from files with jev decide
- Running LLM yes/no checks in GitHub Actions
- Securing a local LLM server with an API key
- curl examples for a local LLM decision API
- Self-hosted AI for decisions
- A private LLM for text classification
- On-premise LLM for business decisions
- GDPR and automated decision-making with an LLM
- Offline AI for decisions: no network needed
- Edge AI decisions on a CPU
- Run an LLM locally without a GPU
- Small language models explained
- When a small model is enough, and when it is not
- An LLM on a laptop: what it can do in real time
- What is GGUF, for someone deploying a classifier
- GGUF quantization types explained: Q4_K_M, Q8_0 and others
- GGUF vs safetensors
- llama.cpp vs Ollama for a classification service
- llama-cpp-python vs calling llama.cpp through ctypes
- llama.cpp on Windows without compiling
- Running llama.cpp CPU only
- Using llama.cpp prebuilt binaries instead of building