-
Notifications
You must be signed in to change notification settings - Fork 132
private llm for text classification
A private LLM for text classification is one that reads the text on a machine you control,
so the text is never sent to a model provider. With jevos the server runs on 127.0.0.1, the
customer message goes in as state, each label is a yes/no question, and what comes back is one
probability per label. No copy of the message is sent anywhere for the model to read it. That
removes one party from your data flow; it does not make the rest of your system private.
The distinction matters because "local" is easy to over-read. Customer text can end up in many places around a model call: application logs, debugging dumps, a proxy, a backup, a shared machine. A local model changes none of those.
This page is what "the data never leaves the machine" covers precisely, what it leaves open, how to send less text in the first place, and when a private small model is not the right classifier.
The model call. When your application posts a ticket to POST /v1/systemone, the text travels
over the loopback interface to a process on the same host, is read by the model, and the answer
goes back the same way. There is no network hop to a third party, no provider-side log of your
prompt, and no question of which region the provider processes it in.
It also makes the data flow easy to describe. For a hosted classifier you would list the provider, what it receives, how long it keeps it and under what contract. For a local one the entry for this step is: "classified on our server by a model file we host". If your server is a rented machine, the hosting company is still part of that description.
| Risk | Does a local model help? | What does |
|---|---|---|
| Text sent to a model provider | Yes, it is not sent | nothing more needed |
| Who can call the classifier | No | bind to 127.0.0.1, or set JEV_API_KEY
|
| Text in application logs | No | log hashes and probabilities, not bodies |
| Text in a reverse proxy or APM tool | No | disable body capture for this route |
| Retention of the messages themselves | No | your retention policy |
| A wrong classification about a person | No | thresholds, review, logging |
Three of those deserve a sentence each.
Access. The server listens on 127.0.0.1 by default, so only processes on the same host
can reach it. If you bind it to a network interface without JEV_API_KEY, anyone who can reach
the port can send text and read answers. With the key set, every call except /health needs a
Bearer token; see securing a local LLM server with an API key.
Logs. The probabilities are not personal text, but they are information about a person ("P(yes) 0.91 that this customer is threatening to cancel"). A decision log that stores the state hash, the question names and the probabilities lets you reconstruct what happened without keeping the message twice. What to keep is on logging LLM decisions for audit.
Decisions. Where a label triggers something that affects the person, the legal questions are the same as with any model. The reading of GDPR Article 22 on GDPR and automated decision-making with an LLM applies to a local model exactly as to a hosted one.
The most private field is the one you never send. state accepts a JSON object, which makes it
easy to pass only what the questions need:
{
"model": "jev-latest",
"state": {
"channel": "email",
"customer_message": "I was charged twice for the same order and nobody answers my emails."
},
"questions": {
"billing": {"type": "noul", "instructions": "Is this a billing problem?"},
"cancel": {"type": "noul", "instructions": "Does the customer say they want to cancel?"},
"repeated": {"type": "noul", "instructions": "Does the customer say they already contacted support?"}
}
}No name, no email address, no order number, no account ID. None of the three questions needs them, and the answers come back keyed by question name, so your code joins them to the customer record it already has. Dropping fields also saves time: latency grows with the length of the input: on the reference laptop a 30-token request read from scratch took 26 ms and a 191-token one 112 ms. The general method is on sending JSON as the text: designing the state.
If messages routinely contain identifiers you would rather not pass at all, strip the exact patterns (emails, phone numbers, card numbers) with a deterministic tool before the model sees the text. That is the split described on checking text for personal data with yes/no questions: exact patterns in code, meaning in the model.
Privacy is worth little if the labels are wrong. The measurements that matter for classification come from 999 yes/no questions written after training, on texts such as emails, tickets, logs, reviews and forms: 0.954 on facts stated in the text, 0.938 on tone, 0.859 on the writer's intent, 0.858 on negation. The weak spots are questions that need a computation, 0.584 on arithmetic and 0.598 on dates, which a classifier mostly should not be asking anyway.
The label design is on zero-shot text classification with yes/no questions: one question per label, several labels in one request, the text read once. On the README's example, three questions take about 66 ms together against 49 ms for one alone.
- Other languages. jevos reads English only. Translating first means sending the text to a translator, which may undo the point of running locally.
- Stable labels with lots of history. If you have thousands of labelled tickets for a label set that does not change, a classifier trained on those examples will usually be more accurate; see a yes/no LLM vs a fine-tuned BERT classifier. It can be just as private.
- Hard rules. On 2,000 questions about unseen business policies, jevos was right 0.810 of the time against 0.927 for the hosted Jev. If the label is really a policy decision, compute the policy in code.
Does a local LLM send data anywhere? The jevos server reads the text in its own process on your machine and sends it nowhere. Downloading the binary and the model needs the network once.
Is a local model private by default? The model call is. Your logs, proxies, backups and who can reach the port are separate decisions.
Can I classify customer emails without a cloud API? Yes, in English, with one yes/no question per label and a threshold on each probability.
Should I remove personal data before classifying? Send only the fields the questions need, and strip exact identifiers in code when the questions do not depend on them.
Is the output personal data? Treat it as such when it is about an identifiable person: a probability that a customer is angry is information about that customer.
See also: self-hosted AI for decisions, offline AI for decisions and email triage with a local LLM.
- Server binding,
JEV_API_KEY, the request format and the three-question timing: the jev README. - Accuracy by kind of question: our 999-question test set on
jevos-q4_k_m; policy accuracy: our 2,000-question comparison. Latency by text length: our measurement on an Intel Core Ultra 7 255H.
From the notes of jev, whose server listens on 127.0.0.1
unless you tell it otherwise.
- Ask a local LLM a yes/no question and get P(yes)
- Zero-shot text classification with yes/no questions
- LLM policy decisions: put the rule in the question
- LLM as a judge on a CPU
- Why a small LLM says yes when the answer is no
- Small LLMs and arithmetic in yes/no questions
- Our held-out benchmark said 0.855, new questions said 0.757
- jevos vs Jev vs Laya for yes/no decisions
- An open-source alternative to Jev for yes/no decisions
- jevos vs the OpenAI API for yes/no classification
- jevos vs Ollama for yes/no decisions
- jevos vs bart-large-mnli for zero-shot classification
- A yes/no LLM vs a fine-tuned BERT classifier
- jevos vs SetFit: zero-shot vs few-shot classification
- jevos vs Llama Guard for content safety checks
- jev serve vs llama.cpp server for classification
- jevos vs LM Studio: a decision server, not a chat app
- Local vs hosted LLM decisions: latency, cost, privacy
- A yes/no LLM vs a business rules engine
- LLM decisions vs keyword rules and regex
- The fastest AI model for yes/no decisions
- What makes a local LLM fast on a CPU
- Why one forward pass beats generating an answer
- Prefill vs decode: where LLM latency comes from
- Why LLM latency grows with the length of the text
- Why a hosted LLM API cannot answer in 50 ms
- Many questions about one text: why the extra ones are cheap
- CPU or GPU for a small LLM
- Latency budgets: where a 200 ms model fits
- Measuring LLM latency: median, p90 and warm-up
- Q4_K_M vs Q8_0: speed and size for a small model
- Throughput vs latency for a decision server
- What P(yes) means, and what it does not
- LLM calibration explained with yes/no answers
- Expected calibration error (ECE), explained
- Temperature scaling for LLM probabilities
- Platt scaling for a yes/no model
- Reading a reliability diagram
- How to choose a threshold for P(yes)
- Thresholds when a wrong yes costs more than a wrong no
- Human in the loop AI with a review band
- Precision and recall at a P(yes) threshold
- Base rates: why a 0.9 yes can still be wrong often
- Combining yes/no answers with AND, OR and NOT
- Logits, log-odds and P(yes)
- LLM confidence scores: probabilities vs self-reports
- How to write yes/no questions an LLM answers well
- Negation in yes/no questions for an LLM
- One condition per question: splitting compound questions
- Ask whether the text says it at all
- Scores as yes/no thresholds: is it at least high?
- Sending JSON as the text: designing the state
- Why wording changes an LLM's answer, and how to test it
- Mainly about: questions for messages with several topics
- Yes/no questions about tone and emotion
- Asking about intent: what does the writer want?
- Yes/no questions about long documents
- Using an English-only LLM with other languages
- Content moderation with a local LLM
- A Discord moderation bot with a local LLM
- Spam detection with yes/no questions
- Review moderation with a local LLM
- Email triage with a local LLM
- Support ticket routing with yes/no questions
- Urgency detection in customer messages
- Sentiment analysis with yes/no questions
- Intent detection with a local LLM
- Lead qualification with yes/no questions
- Fraud case triage with a local LLM
- Phishing email screening with a local LLM
- Log and alert triage with a local LLM
- Checking text for personal data with yes/no questions
- Prompt injection screening with a small model
- Document classification with a local LLM
- Product categorization with yes/no questions
- Contract clause detection with a local LLM
- Refund request triage with a local LLM
- Detecting cancellation intent in customer messages
- RAG evaluation with yes/no questions
- RAG faithfulness check with a local LLM
- Hallucination detection with a local LLM
- LLM regression tests in CI with yes/no checks
- Rubric design for an LLM judge
- Pairwise comparison with a yes/no judge
- LLM judge bias and how to control it
- Evaluation metrics for yes/no classifiers
- Building a yes/no test set for your own data
- Accuracy by kind of question: why one number hides failures
- Generating test questions with answers computed by code
- Benchmark contamination and truly held-out tests
- An LLM router with yes/no questions
- A model cascade: small model first, large model on doubt
- Semantic routing vs yes/no questions
- Gating AI agent tool calls with yes/no checks
- AI agent guardrails with yes/no questions
- Stop conditions for AI agents
- Logging LLM decisions for audit
- Reducing LLM cost with local yes/no decisions
- Replacing chat LLM calls with yes/no questions
- Structured output vs a probability
- A Python client for local LLM decisions
- Calling a local LLM decision server from JavaScript
- Local LLM yes/no decisions in n8n
- A Slack bot that uses local LLM decisions
- Home Assistant automations with local LLM decisions
- A LangChain tool for local yes/no decisions
- Batch decisions from files with jev decide
- Running LLM yes/no checks in GitHub Actions
- Securing a local LLM server with an API key
- curl examples for a local LLM decision API
- Self-hosted AI for decisions
- A private LLM for text classification
- On-premise LLM for business decisions
- GDPR and automated decision-making with an LLM
- Offline AI for decisions: no network needed
- Edge AI decisions on a CPU
- Run an LLM locally without a GPU
- Small language models explained
- When a small model is enough, and when it is not
- An LLM on a laptop: what it can do in real time
- What is GGUF, for someone deploying a classifier
- GGUF quantization types explained: Q4_K_M, Q8_0 and others
- GGUF vs safetensors
- llama.cpp vs Ollama for a classification service
- llama-cpp-python vs calling llama.cpp through ctypes
- llama.cpp on Windows without compiling
- Running llama.cpp CPU only
- Using llama.cpp prebuilt binaries instead of building