Repository navigation
latency budgets for llm decisions
A decision that takes 30 to 130 ms fits almost everything except work that has to finish inside a single animation frame. It is well inside a webhook timeout of a few seconds, fast enough to feel immediate behind a button, and cheap enough to run on every item of a batch job. It does not fit a real-time loop at 60 frames per second, or a user interface that must react within a tenth of a second to every keystroke. Those are the two edges of the budget.
The useful habit is to treat the model as one line of a budget, not the whole of it. The request has to be built, sent, answered and acted on, and a 200 ms model inside a 250 ms budget leaves nothing for the rest.
This page is the published limits people design to, how a 30 to 130 ms model sits in each context, how to add up a budget, and what to do when it does not fit.
| Context | Budget | Source |
|---|---|---|
| Feels instantaneous | 0.1 s | Nielsen, response time limits |
| Flow of thought uninterrupted | 1 s | Nielsen, response time limits |
| Attention kept on the task | 10 s | Nielsen, response time limits |
| Visible response to input | 100 ms, process input within 50 ms | Google's RAIL model |
| One animation frame at 60 fps | about 16 ms, aim for 10 ms | Google's RAIL model |
| Slack event webhook | respond within 3 s | Slack Events API |
Nielsen's three limits date from 1993: 0.1 second is "the limit for having the user feel that the system is reacting instantaneously", 1 second "for the user's flow of thought to stay uninterrupted". RAIL asks web pages to process input within 50 ms so a visible response can happen within 100 ms.
jevos, on our reference laptop (Intel Core Ultra 7 255H, 16 threads, 8-bit weights), took 28 ms on a short request and 130 ms on a long one, each read from scratch (median of 10 requests after 3 warm-up requests, measured with jevos-v3, which has the same size and speed as jevos-v4). That puts short requests well under the "instantaneous" line and long ones just past it, well inside the one-second line.
A user clicks "submit" on a support form and the app decides where the ticket goes. Under 1 second the user's flow is not broken, so a 130 ms decision fits with room for the rest of the request. What fits less well is running the model on every keystroke to give live feedback: a short text, at 28 ms of model time plus the round trip, fits the 100 ms target, and a long text, at 130 ms, misses it.
For interactive use, show something immediately and let the decision arrive a moment later. Keep the state short: on our laptop a short request took 28 ms and a long one 130 ms; the detail is on why LLM latency grows with the length of the text.
Event senders give you a deadline. Slack's Events API says an app "should respond to the event request with an HTTP 2xx within three seconds", and retries if it does not. A 130 ms decision fits many times over. Slack also asks apps to respond "as soon as you can", so acknowledging first and deciding after is still the safer design: a slow moment elsewhere does not cause retries.
Chat moderation fits the same way: a message arrives, the rules are asked as questions in one request, and the bot acts before most people have read the message. That architecture is sketched on a Discord moderation bot with a local LLM, and for inboxes on email triage with a local LLM.
Here latency becomes cost per item. At 130 ms per long record, one process handles about seven records per second when requests run one after another; at 130 ms each a million records would take about 36 hours on one process, which is arithmetic, not a measured run. The budget is the length of the job window, not a reaction time. For files on disk, batch decisions with jev decide covers running requests without a server, and how capacity differs from latency is on throughput vs latency for a decision server.
Two levers matter most in batch: grouping questions about the same record in one request (three questions took about 66 ms together against 49 ms for one alone, see many questions about one text), and trimming each record to the fields the questions need.
This is where a 30 to 130 ms model does not fit. At 60 frames per second a frame is about 16 ms, and RAIL suggests 10 ms of work per frame. Even the short request is longer than a frame.
The honest way around it is to let the loop wait for the model and show the time per decision, which demonstrates decisions, not reaction speed. A real-time system would ask the model less often, about slower-changing things (a strategy, not a jump), and keep the per-frame logic in code.
An illustrative example for a web request with a 500 ms target, with made-up numbers for everything except the model:
| Step | Time |
|---|---|
| Receive request, load the record | 30 ms (illustrative) |
| Build the state, trim fields | 5 ms (illustrative) |
| Model, one request, three questions | about 66 ms (measured on our laptop) |
| Act on thresholds, write a log line | 20 ms (illustrative) |
| Total | about 120 ms |
That leaves room for a slow moment, and for the difference between a median and a p90: budgets should be set against the slow end, not the typical request. How to measure that end is on measuring LLM latency: median, p90 and warm-up.
When it does not fit: shorten the input, group the questions, move the decision off the critical path (decide after responding), or ask less often. A hosted model adds a round trip on top of all of this, which is why it rarely fits the tight end; see why a hosted LLM API cannot answer in 50 ms.
Is 200 ms fast enough for a user interface? For an action like submitting a form, yes: it is well under the 1-second limit for uninterrupted flow. For reacting to every keystroke, it is tight.
Can a local LLM run inside a game loop? Not per frame. At 60 fps a frame is about 16 ms. Ask the model about slower decisions and keep per-frame logic in code.
Does a 130 ms model fit a webhook? Yes. Slack, for example, allows three seconds.
How many decisions per second is 130 ms? About seven, if requests run one after another on one process (arithmetic, not a measured run). Capacity under concurrency is a different measurement that is not published for jevos-v4.
What should I budget against, the median or the p90? The slow end. A budget met by the median is missed by one request in two.
See also: the fastest AI model for yes/no decisions, an LLM on a laptop and AI agent guardrails with yes/no questions.
- jevos latencies (28 ms, 130 ms, 66 ms against 49 ms): our measurements with jevos-v3 (same size and speed as jevos-v4) and the jev README.
- Response time limits: Jakob Nielsen, Response Times: The 3 Important Limits, 1993, fetched 2026-09-29.
- Input and frame budgets: RAIL model, web.dev, fetched 2026-09-29.
- Webhook deadline: Slack Events API, fetched 2026-09-29.
- Rows marked illustrative in the budget table are made up for the example.
From the notes of jev, whose own game demo pauses for the model, because we would rather show the real decision time than hide it behind a frame rate.
- Ask a local LLM a yes/no question and get P(yes)
- Zero-shot text classification with yes/no questions
- LLM policy decisions: put the rule in the question
- LLM as a judge on a CPU
- Why a small LLM says yes when the answer is no
- Small LLMs and arithmetic in yes/no questions
- Our held-out benchmark said 0.855, new questions said 0.757
- Does an LLM know when a fact is missing?
- jevos vs Jev vs Laya for yes/no decisions
- An open-source alternative to Jev for yes/no decisions
- jevos vs the OpenAI API for yes/no classification
- jevos vs Ollama for yes/no decisions
- jevos vs bart-large-mnli for zero-shot classification
- A yes/no LLM vs a fine-tuned BERT classifier
- jevos vs SetFit: zero-shot vs few-shot classification
- jevos vs Llama Guard for content safety checks
- jev serve vs llama.cpp server for classification
- jevos vs LM Studio: a decision server, not a chat app
- Local vs hosted LLM decisions: latency, cost, privacy
- A yes/no LLM vs a business rules engine
- LLM decisions vs keyword rules and regex
- The fastest AI model for yes/no decisions
- What makes a local LLM fast on a CPU
- Why one forward pass beats generating an answer
- Prefill vs decode: where LLM latency comes from
- Why LLM latency grows with the length of the text
- Why a hosted LLM API cannot answer in 50 ms
- Many questions about one text: why the extra ones are cheap
- CPU or GPU for a small LLM
- Latency budgets: where a 200 ms model fits
- Measuring LLM latency: median, p90 and warm-up
- Q4_K_M vs Q8_0: speed and size for a small model
- Throughput vs latency for a decision server
- What P(yes) means, and what it does not
- LLM calibration explained with yes/no answers
- Expected calibration error (ECE), explained
- Temperature scaling for LLM probabilities
- Platt scaling for a yes/no model
- Reading a reliability diagram
- How to choose a threshold for P(yes)
- Thresholds when a wrong yes costs more than a wrong no
- Human in the loop AI with a review band
- Precision and recall at a P(yes) threshold
- Base rates: why a 0.9 yes can still be wrong often
- Combining yes/no answers with AND, OR and NOT
- Logits, log-odds and P(yes)
- LLM confidence scores: probabilities vs self-reports
- How to write yes/no questions an LLM answers well
- Negation in yes/no questions for an LLM
- One condition per question: splitting compound questions
- Ask whether the text says it at all
- Scores as yes/no thresholds: is it at least high?
- Sending JSON as the text: designing the state
- Why wording changes an LLM's answer, and how to test it
- Mainly about: questions for messages with several topics
- Yes/no questions about tone and emotion
- Asking about intent: what does the writer want?
- Yes/no questions about long documents
- Using an English-only LLM with other languages
- Content moderation with a local LLM
- A Discord moderation bot with a local LLM
- Spam detection with yes/no questions
- Review moderation with a local LLM
- Email triage with a local LLM
- Support ticket routing with yes/no questions
- Urgency detection in customer messages
- Sentiment analysis with yes/no questions
- Intent detection with a local LLM
- Lead qualification with yes/no questions
- Fraud case triage with a local LLM
- Phishing email screening with a local LLM
- Log and alert triage with a local LLM
- Checking text for personal data with yes/no questions
- Prompt injection screening with a small model
- Document classification with a local LLM
- Product categorization with yes/no questions
- Contract clause detection with a local LLM
- Refund request triage with a local LLM
- Detecting cancellation intent in customer messages
- RAG evaluation with yes/no questions
- RAG faithfulness check with a local LLM
- Hallucination detection with a local LLM
- LLM regression tests in CI with yes/no checks
- Rubric design for an LLM judge
- Pairwise comparison with a yes/no judge
- LLM judge bias and how to control it
- Evaluation metrics for yes/no classifiers
- Building a yes/no test set for your own data
- Accuracy by kind of question: why one number hides failures
- Generating test questions with answers computed by code
- Benchmark contamination and truly held-out tests
- An LLM router with yes/no questions
- A model cascade: small model first, large model on doubt
- Semantic routing vs yes/no questions
- Gating AI agent tool calls with yes/no checks
- AI agent guardrails with yes/no questions
- Stop conditions for AI agents
- Logging LLM decisions for audit
- Reducing LLM cost with local yes/no decisions
- Replacing chat LLM calls with yes/no questions
- Structured output vs a probability
- A Python client for local LLM decisions
- Calling a local LLM decision server from JavaScript
- Local LLM yes/no decisions in n8n
- A Slack bot that uses local LLM decisions
- Home Assistant automations with local LLM decisions
- A LangChain tool for local yes/no decisions
- Batch decisions from files with jev decide
- Running LLM yes/no checks in GitHub Actions
- Securing a local LLM server with an API key
- curl examples for a local LLM decision API
- Self-hosted AI for decisions
- A private LLM for text classification
- On-premise LLM for business decisions
- GDPR and automated decision-making with an LLM
- Offline AI for decisions: no network needed
- Edge AI decisions on a CPU
- Run an LLM locally without a GPU
- Small language models explained
- When a small model is enough, and when it is not
- An LLM on a laptop: what it can do in real time
- What is GGUF, for someone deploying a classifier
- GGUF quantization types explained: Q4_K_M, Q8_0 and others
- GGUF vs safetensors
- llama.cpp vs Ollama for a classification service
- llama-cpp-python vs calling llama.cpp through ctypes
- llama.cpp on Windows without compiling
- Running llama.cpp CPU only
- Using llama.cpp prebuilt binaries instead of building