-
Notifications
You must be signed in to change notification settings - Fork 129
jevos vs openai api for classification
A hosted OpenAI model answers a yes/no question by generating an answer, which you parse or read the token probabilities of; jevos answers it on your own CPU with a probability and no text at all. The OpenAI API is the stronger reader, handles many languages and can explain itself. jevos avoids the network round trip, the per-token bill and sending the text anywhere, and it has no output format to break. If the question is hard, or not in English, the API is the better tool; if it is one of many simple reading questions about English text, a local probability is usually enough.
Conflict of interest, in one line: we build jevos, and every OpenAI fact on this page comes from OpenAI's own documentation, fetched 2026-09-29.
The difference is not only speed. A generated "Yes" is a string your code must interpret, and a probability is a number your code can threshold. That changes how you handle doubt: with a probability, "not sure" is a range you can route to a person instead of a parsing edge case.
This page is the three ways to get a yes or no from the API, what each one leaves you to parse, where latency comes from, what you pay for, where the text goes, and when to prefer the API.
Ask for one word. "Answer yes or no." The model generates a token or a few, and your code compares the text with "yes". It works most of the time, and the rest of the time you get "Yes.", "yes, because..." or a refusal, and you write the normalisation.
Ask for a schema. OpenAI's Structured Outputs "ensures the model will always generate
responses that adhere to your supplied JSON Schema", so a field such as {"answer": boolean}
arrives well formed. The same guide lists the exceptions: a safety refusal, reported in a
separate refusal field, and an incomplete response when a token limit is reached. You get a
clean boolean, but still no measure of doubt.
Read the logprobs. With logprobs enabled the API returns the log probability of each output
token, and top_logprobs (an integer from 0 to 5) adds the most likely alternatives at each
position. OpenAI's cookbook uses this for classification: read the probability of the label the
model chose, send results above a threshold through automatically and the less certain ones to
manual review. This is the closest thing to P(yes), with two caveats from the same cookbook:
logprobs are for output tokens, so the model still has to generate at least one, and you have to
find the "yes" and "no" tokens among the alternatives yourself.
| what comes back | what your code does | |
|---|---|---|
| one-word answer | text | normalise and match, handle surprises |
| Structured Outputs | JSON matching your schema | check for refusal or an incomplete status |
| logprobs | tokens with log probabilities | find the yes and no tokens, convert, renormalise |
| jevos |
{"noul": 0.9} per question |
compare with a threshold |
The last row is not a claim that jevos is smarter. It is that nothing is generated
(output_tokens is always 0), so there is no text to be malformed. The general argument is on
structured output vs a probability.
OpenAI's latency guide says that generating tokens "is almost always the highest latency step when using an LLM" and that each request adds round-trip latency. A one-word answer keeps generation short, but the round trip stays: DNS, TLS, the request crossing the network, queueing at the provider, and the response coming back.
We have not measured OpenAI's API, so there is no OpenAI number on this page. What we have measured is a different hosted decision API from a laptop in Europe: TypeSafe's Jev took 344 ms on a short request and 345 ms on a long one, network included, while jevos took 26 ms and 112 ms on the same laptop. The flatness of the hosted numbers is the point: when the network dominates, text length barely matters. The detail is on why a hosted LLM API cannot answer in 50 ms.
Batching several questions into one request is the fix both sides share. OpenAI's guide suggests putting sequential steps into a single prompt to avoid extra round trips; jevos reads the state once for every question in a request, so three questions took about 66 ms against 49 ms for one.
OpenAI prices its models per million tokens, with input, cached input and output listed as separate rates. A yes/no call pays for the whole prompt on every request, the text plus the instructions, and for the output tokens, however few.
jevos has no per-token price. The cost is the machine it runs on: a CPU and the memory for the loaded model. At low volume the API is cheaper than keeping a server up; at high volume on short decisions the arithmetic tends to flip. Where that line falls depends on your volume and your prices, which is why reducing LLM cost with local yes/no decisions starts with counting how many of your calls are really decisions.
OpenAI's data controls page states that data sent to the API is not used to train its models unless you opt in, and that abuse monitoring logs are retained for up to 30 days unless longer retention is required by law. Zero Data Retention and data residency in a list of regions exist, subject to OpenAI's approval.
With jevos the text does not leave the machine, because the server listens on 127.0.0.1 by default and answers without a network call. That removes a data processor from the picture. It does not remove your own obligations: logs, access control and retention on your side are still yours, as a private LLM for text classification explains.
- The question needs reasoning or knowledge beyond the text, or arithmetic. jevos was right 0.584 of the time on arithmetic questions in our 999-question test, against 0.954 on facts stated in the text.
- The text is not in English. jevos reads English only.
- You need a label with an explanation, or free text, alongside the decision.
- Your volume is low and you do not want to operate a server.
Can I get a probability from the OpenAI API? Partly: logprobs and top_logprobs return log
probabilities of the output tokens, so you can read how likely the generated "yes" was.
Does Structured Outputs solve parsing? It gives a well-formed JSON value, except on refusals and incomplete responses, but a boolean carries no measure of doubt.
Is a local model faster than the API? We did not measure OpenAI. Against a hosted decision API from Europe, jevos was faster on both of our requests: 26 and 112 ms against 344 and 345 ms.
Is it cheaper? There is no per-token price locally; you pay for the hardware. Whether that is cheaper depends on volume.
Is jevos as accurate as a large hosted model? No, especially on computation. It is strongest on reading questions: facts, tone, negation.
See also: local vs hosted LLM decisions, jevos vs Jev vs Laya and LLM confidence scores: probabilities vs self-reported confidence.
- Our measurements: latency of jevos and of TypeSafe's Jev on the same two requests, the
three-question timing, and accuracy by kind on our 999-question set (
jevos-q4_k_m); see the jev README. - OpenAI, Structured model outputs, fetched 2026-09-29.
- OpenAI cookbook, Using logprobs, fetched 2026-09-29.
- OpenAI, Latency optimization, fetched 2026-09-29.
- OpenAI, Data controls in the OpenAI platform, fetched 2026-09-29.
- OpenAI, API pricing, for the per-million-token structure only, fetched 2026-09-29.
From the notes of jev, a yes/no decision model that runs on a laptop CPU and has no OpenAI number of its own to report, so it reports none.
- Ask a local LLM a yes/no question and get P(yes)
- Zero-shot text classification with yes/no questions
- LLM policy decisions: put the rule in the question
- LLM as a judge on a CPU
- Why a small LLM says yes when the answer is no
- Small LLMs and arithmetic in yes/no questions
- Our held-out benchmark said 0.855, new questions said 0.757
- jevos vs Jev vs Laya for yes/no decisions
- An open-source alternative to Jev for yes/no decisions
- jevos vs the OpenAI API for yes/no classification
- jevos vs Ollama for yes/no decisions
- jevos vs bart-large-mnli for zero-shot classification
- A yes/no LLM vs a fine-tuned BERT classifier
- jevos vs SetFit: zero-shot vs few-shot classification
- jevos vs Llama Guard for content safety checks
- jev serve vs llama.cpp server for classification
- jevos vs LM Studio: a decision server, not a chat app
- Local vs hosted LLM decisions: latency, cost, privacy
- A yes/no LLM vs a business rules engine
- LLM decisions vs keyword rules and regex
- The fastest AI model for yes/no decisions
- What makes a local LLM fast on a CPU
- Why one forward pass beats generating an answer
- Prefill vs decode: where LLM latency comes from
- Why LLM latency grows with the length of the text
- Why a hosted LLM API cannot answer in 50 ms
- Many questions about one text: why the extra ones are cheap
- CPU or GPU for a small LLM
- Latency budgets: where a 200 ms model fits
- Measuring LLM latency: median, p90 and warm-up
- Q4_K_M vs Q8_0: speed and size for a small model
- Throughput vs latency for a decision server
- What P(yes) means, and what it does not
- LLM calibration explained with yes/no answers
- Expected calibration error (ECE), explained
- Temperature scaling for LLM probabilities
- Platt scaling for a yes/no model
- Reading a reliability diagram
- How to choose a threshold for P(yes)
- Thresholds when a wrong yes costs more than a wrong no
- Human in the loop AI with a review band
- Precision and recall at a P(yes) threshold
- Base rates: why a 0.9 yes can still be wrong often
- Combining yes/no answers with AND, OR and NOT
- Logits, log-odds and P(yes)
- LLM confidence scores: probabilities vs self-reports
- How to write yes/no questions an LLM answers well
- Negation in yes/no questions for an LLM
- One condition per question: splitting compound questions
- Ask whether the text says it at all
- Scores as yes/no thresholds: is it at least high?
- Sending JSON as the text: designing the state
- Why wording changes an LLM's answer, and how to test it
- Mainly about: questions for messages with several topics
- Yes/no questions about tone and emotion
- Asking about intent: what does the writer want?
- Yes/no questions about long documents
- Using an English-only LLM with other languages
- Content moderation with a local LLM
- A Discord moderation bot with a local LLM
- Spam detection with yes/no questions
- Review moderation with a local LLM
- Email triage with a local LLM
- Support ticket routing with yes/no questions
- Urgency detection in customer messages
- Sentiment analysis with yes/no questions
- Intent detection with a local LLM
- Lead qualification with yes/no questions
- Fraud case triage with a local LLM
- Phishing email screening with a local LLM
- Log and alert triage with a local LLM
- Checking text for personal data with yes/no questions
- Prompt injection screening with a small model
- Document classification with a local LLM
- Product categorization with yes/no questions
- Contract clause detection with a local LLM
- Refund request triage with a local LLM
- Detecting cancellation intent in customer messages
- RAG evaluation with yes/no questions
- RAG faithfulness check with a local LLM
- Hallucination detection with a local LLM
- LLM regression tests in CI with yes/no checks
- Rubric design for an LLM judge
- Pairwise comparison with a yes/no judge
- LLM judge bias and how to control it
- Evaluation metrics for yes/no classifiers
- Building a yes/no test set for your own data
- Accuracy by kind of question: why one number hides failures
- Generating test questions with answers computed by code
- Benchmark contamination and truly held-out tests
- An LLM router with yes/no questions
- A model cascade: small model first, large model on doubt
- Semantic routing vs yes/no questions
- Gating AI agent tool calls with yes/no checks
- AI agent guardrails with yes/no questions
- Stop conditions for AI agents
- Logging LLM decisions for audit
- Reducing LLM cost with local yes/no decisions
- Replacing chat LLM calls with yes/no questions
- Structured output vs a probability
- A Python client for local LLM decisions
- Calling a local LLM decision server from JavaScript
- Local LLM yes/no decisions in n8n
- A Slack bot that uses local LLM decisions
- Home Assistant automations with local LLM decisions
- A LangChain tool for local yes/no decisions
- Batch decisions from files with jev decide
- Running LLM yes/no checks in GitHub Actions
- Securing a local LLM server with an API key
- curl examples for a local LLM decision API
- Self-hosted AI for decisions
- A private LLM for text classification
- On-premise LLM for business decisions
- GDPR and automated decision-making with an LLM
- Offline AI for decisions: no network needed
- Edge AI decisions on a CPU
- Run an LLM locally without a GPU
- Small language models explained
- When a small model is enough, and when it is not
- An LLM on a laptop: what it can do in real time
- What is GGUF, for someone deploying a classifier
- GGUF quantization types explained: Q4_K_M, Q8_0 and others
- GGUF vs safetensors
- llama.cpp vs Ollama for a classification service
- llama-cpp-python vs calling llama.cpp through ctypes
- llama.cpp on Windows without compiling
- Running llama.cpp CPU only
- Using llama.cpp prebuilt binaries instead of building