-
Notifications
You must be signed in to change notification settings - Fork 132
local vs hosted llm decisions
A local model wins on latency for short decisions, costs nothing per call and keeps the text on your machine; a hosted model wins on accuracy for hard questions, on languages, and on not having to run anything. In our one measured example, from a laptop in Europe, jevos answered a short request in 26 ms and a long one in 112 ms, while TypeSafe's hosted Jev took 344 and 345 ms with the network included. On 2,000 rule questions, the hosted model was right 0.927 of the time against 0.810 locally. Which of those rows matters most is the decision.
Conflict of interest, in one line: we build jevos, the local side of the one measurement on this page.
The non-obvious part is that the two latency curves have different shapes. Local time grows with the length of the text; hosted time is dominated by a fixed round trip and barely moves. So "local is faster" is true for short inputs and can stop being true for long ones, and where the lines cross depends on your region, your text and your hardware.
This page is the measurement, then latency, cost, privacy and accuracy one at a time, then what changes with scale and region.
| short request (about 30 tokens) | long request (about 190 tokens) | |
|---|---|---|
| jevos, local (Intel Core Ultra 7 255H, 16 threads, CPU) | 26 ms | 112 ms |
| TypeSafe Jev, hosted API from Europe, network included | 344 ms | 345 ms |
Same two requests, same laptop. The hosted numbers include everything an application would see: connection, request, the provider's processing and the response. It is one hosted service from one place, not a statement about hosted APIs in general, and a server closer to the provider would see less.
The local time is almost all prompt processing, and no generation, since jevos produces no output tokens. More text takes longer: 26 ms for about 30 tokens, 112 ms for about 190 on that laptop, unless the same text was read before (22 ms for the long one asked again). The mechanics are on why LLM latency grows with the length of the text.
The hosted time has a floor that has nothing to do with the model: DNS, TLS, the distance to the data centre, queueing. OpenAI's latency guide makes the general point that "each time you make a request, you incur some round-trip latency" and recommends fewer, combined requests. Above the floor, a fast hosted model adds little per token, which is why Jev's two numbers are 1 ms apart.
Two consequences follow. For short, interactive decisions, such as a chat message, a form field or a game loop, the local slope stays well under the hosted floor. For long documents the local time keeps growing and the hosted floor does not, so past some length, which we have not measured, the hosted call can be the faster one if your network is good.
A hosted API bills per token, with input and output priced separately; OpenAI's pricing page, for example, lists rates per million tokens. A yes/no call pays for the text and the instructions every time. The bill scales linearly with volume, and there is nothing to operate.
A local model has no per-call price. You pay for a machine with a CPU and about 1 GB of free memory for the loaded model, and for the time to operate it. At low volume that is more expensive than the API; at high volume on short decisions it tends to be much cheaper. Grouping questions helps both sides: jevos reads the text once for every question in a request, so three questions took about 66 ms against 49 ms for one.
Local means the text is not sent to a third party. The jev server listens on 127.0.0.1 by default,
and with JEV_API_KEY set every call except /health needs a bearer token. For customer
messages, health data or contracts, removing a processor from the chain can remove a lot of
paperwork.
Hosted can be acceptable too. OpenAI's data controls page, as one example, states that API data is not used for training unless you opt in, that abuse monitoring logs are kept for up to 30 days, and that zero data retention and regional storage are available with approval. Read your provider's terms, not a summary.
What local does not solve: who can reach the server, what your own logs keep, and how long. See a private LLM for text classification.
On 2,000 yes/no questions about three business policies none of the models had been tuned on, Jev was right 0.927 of the time and jevos 0.810. The gap was largest on additive point scores, several signals summed and compared with a cut-off, which is arithmetic. On reading questions the local model is much stronger than on computation: 0.954 on stated facts against 0.584 on arithmetic in our 999-question test.
If a wrong decision is expensive and the questions involve rules and numbers, the hosted model's extra accuracy may be worth all its latency and cost. If the questions are reading questions, the local one may be enough. The way to know is a labelled sample of your own cases.
- Volume. Cost favours local as volume grows; the break-even depends on your prices and hardware. Capacity is a separate question from the latency above, which was measured one request at a time; see throughput vs latency for a decision server.
- Region. A client near the provider sees a lower floor. A client far away sees a higher one.
- Text length. Short texts favour local; very long ones narrow or reverse the gap.
-
Languages and question types. jevos is English, and answers yes/no and
choicequestions; itsscoreanswers are early (54% on held-out score questions, 82% within one level). - Both at once. With the same wire format, you can answer confident cases locally and send the uncertain band to the hosted model, as on a model cascade: small model first.
Is a local LLM faster than an API? For short decisions in our measurement, yes: 26 ms against 344 ms. For long texts the gap narrows, because local time grows with length and hosted time barely does.
Is a local LLM cheaper? No per-call price, but you pay for and run the machine. At high volume it usually is; at low volume the API often is.
Is a local LLM more private? The text stays on your machine. Access control and logs remain your job.
Is hosted more accurate? In our test, yes: 0.927 against 0.810 on unseen rules.
Can I use both? Yes. Decide locally when confident and escalate the rest.
See also: why a hosted LLM API cannot answer in 50 ms, jevos vs the OpenAI API for yes/no classification and jevos vs Jev vs Laya.
- Latency (26/112 ms, 344/345 ms), the three-question timing, memory and the 2,000-question accuracy: our own measurements, published in the jev README.
- Accuracy by kind: our 999-question test set on
jevos-q4_k_m. - Round-trip latency advice: OpenAI, Latency optimization, fetched 2026-09-29.
- Per-million-token pricing: OpenAI, API pricing, fetched 2026-09-29.
- Data use, retention and residency: OpenAI, Data controls, fetched 2026-09-29.
From the notes of jev, measured from one laptop in Europe, which is exactly as general as that sounds.
- Ask a local LLM a yes/no question and get P(yes)
- Zero-shot text classification with yes/no questions
- LLM policy decisions: put the rule in the question
- LLM as a judge on a CPU
- Why a small LLM says yes when the answer is no
- Small LLMs and arithmetic in yes/no questions
- Our held-out benchmark said 0.855, new questions said 0.757
- jevos vs Jev vs Laya for yes/no decisions
- An open-source alternative to Jev for yes/no decisions
- jevos vs the OpenAI API for yes/no classification
- jevos vs Ollama for yes/no decisions
- jevos vs bart-large-mnli for zero-shot classification
- A yes/no LLM vs a fine-tuned BERT classifier
- jevos vs SetFit: zero-shot vs few-shot classification
- jevos vs Llama Guard for content safety checks
- jev serve vs llama.cpp server for classification
- jevos vs LM Studio: a decision server, not a chat app
- Local vs hosted LLM decisions: latency, cost, privacy
- A yes/no LLM vs a business rules engine
- LLM decisions vs keyword rules and regex
- The fastest AI model for yes/no decisions
- What makes a local LLM fast on a CPU
- Why one forward pass beats generating an answer
- Prefill vs decode: where LLM latency comes from
- Why LLM latency grows with the length of the text
- Why a hosted LLM API cannot answer in 50 ms
- Many questions about one text: why the extra ones are cheap
- CPU or GPU for a small LLM
- Latency budgets: where a 200 ms model fits
- Measuring LLM latency: median, p90 and warm-up
- Q4_K_M vs Q8_0: speed and size for a small model
- Throughput vs latency for a decision server
- What P(yes) means, and what it does not
- LLM calibration explained with yes/no answers
- Expected calibration error (ECE), explained
- Temperature scaling for LLM probabilities
- Platt scaling for a yes/no model
- Reading a reliability diagram
- How to choose a threshold for P(yes)
- Thresholds when a wrong yes costs more than a wrong no
- Human in the loop AI with a review band
- Precision and recall at a P(yes) threshold
- Base rates: why a 0.9 yes can still be wrong often
- Combining yes/no answers with AND, OR and NOT
- Logits, log-odds and P(yes)
- LLM confidence scores: probabilities vs self-reports
- How to write yes/no questions an LLM answers well
- Negation in yes/no questions for an LLM
- One condition per question: splitting compound questions
- Ask whether the text says it at all
- Scores as yes/no thresholds: is it at least high?
- Sending JSON as the text: designing the state
- Why wording changes an LLM's answer, and how to test it
- Mainly about: questions for messages with several topics
- Yes/no questions about tone and emotion
- Asking about intent: what does the writer want?
- Yes/no questions about long documents
- Using an English-only LLM with other languages
- Content moderation with a local LLM
- A Discord moderation bot with a local LLM
- Spam detection with yes/no questions
- Review moderation with a local LLM
- Email triage with a local LLM
- Support ticket routing with yes/no questions
- Urgency detection in customer messages
- Sentiment analysis with yes/no questions
- Intent detection with a local LLM
- Lead qualification with yes/no questions
- Fraud case triage with a local LLM
- Phishing email screening with a local LLM
- Log and alert triage with a local LLM
- Checking text for personal data with yes/no questions
- Prompt injection screening with a small model
- Document classification with a local LLM
- Product categorization with yes/no questions
- Contract clause detection with a local LLM
- Refund request triage with a local LLM
- Detecting cancellation intent in customer messages
- RAG evaluation with yes/no questions
- RAG faithfulness check with a local LLM
- Hallucination detection with a local LLM
- LLM regression tests in CI with yes/no checks
- Rubric design for an LLM judge
- Pairwise comparison with a yes/no judge
- LLM judge bias and how to control it
- Evaluation metrics for yes/no classifiers
- Building a yes/no test set for your own data
- Accuracy by kind of question: why one number hides failures
- Generating test questions with answers computed by code
- Benchmark contamination and truly held-out tests
- An LLM router with yes/no questions
- A model cascade: small model first, large model on doubt
- Semantic routing vs yes/no questions
- Gating AI agent tool calls with yes/no checks
- AI agent guardrails with yes/no questions
- Stop conditions for AI agents
- Logging LLM decisions for audit
- Reducing LLM cost with local yes/no decisions
- Replacing chat LLM calls with yes/no questions
- Structured output vs a probability
- A Python client for local LLM decisions
- Calling a local LLM decision server from JavaScript
- Local LLM yes/no decisions in n8n
- A Slack bot that uses local LLM decisions
- Home Assistant automations with local LLM decisions
- A LangChain tool for local yes/no decisions
- Batch decisions from files with jev decide
- Running LLM yes/no checks in GitHub Actions
- Securing a local LLM server with an API key
- curl examples for a local LLM decision API
- Self-hosted AI for decisions
- A private LLM for text classification
- On-premise LLM for business decisions
- GDPR and automated decision-making with an LLM
- Offline AI for decisions: no network needed
- Edge AI decisions on a CPU
- Run an LLM locally without a GPU
- Small language models explained
- When a small model is enough, and when it is not
- An LLM on a laptop: what it can do in real time
- What is GGUF, for someone deploying a classifier
- GGUF quantization types explained: Q4_K_M, Q8_0 and others
- GGUF vs safetensors
- llama.cpp vs Ollama for a classification service
- llama-cpp-python vs calling llama.cpp through ctypes
- llama.cpp on Windows without compiling
- Running llama.cpp CPU only
- Using llama.cpp prebuilt binaries instead of building