-
Notifications
You must be signed in to change notification settings - Fork 132
designing the state as json
Put in the state only what the questions need, name every field so it reads as English,
and send computed values ("delivered": "5 days ago") instead of raw data the model would have
to calculate from. The state can be a plain string or any JSON object or array, and all of
it is read the first time the model sees it: every field costs time, and every field the model has to
interpret is a place it can go wrong. A state designed for reading is short, plain and already
reduced to facts.
The non-obvious point is that the state is where most computation should disappear. The question "Was it delivered within 30 days?" is only as hard as the state makes it. Two ISO timestamps turn it into a date subtraction, which is the model's weakest skill; "delivered: 5 days ago" turns it into reading.
This page is what to include and leave out, how to name fields, which values to compute before sending, and what size costs.
Everything a question depends on, and nothing it does not. The README's refund example is a good pattern:
{
"model": "jev-latest",
"state": {
"item": "wireless mouse",
"delivered": "5 days ago",
"customer_message": "The box arrived empty. This is the second time!"
},
"questions": {
"refund": {"type": "noul", "instructions": "Our policy refunds items reported missing within 30 days of delivery. Should this customer get a refund?"},
"upset": {"type": "noul", "instructions": "Is the customer upset?"}
}
}Three fields: what was bought, a duration the rule needs, and the customer's own words. The
README reports 95 input tokens for this state with three questions. What is not there matters
as much: no order ID, no warehouse code, no internal status flags, no customer email address.
None of them helps answer any question, all of them cost tokens, and a field such as
"status": "REFUND_PENDING" can nudge the model toward an answer it should have reached from
the message.
If the record carries personal data the questions do not need, leaving it out is also the simpler privacy position, even on a model that runs on your own machine.
Log lines and alerts are the other common JSON state; which of their fields to keep is shown on log and alert triage with a local LLM.
Assume the model reads the keys as well as the values. Name fields the way you would describe them to a new colleague:
| Instead of | Use |
|---|---|
cust_msg |
customer_message |
dlv_ts |
delivered |
amt |
order_total |
flg_rpt |
reported_before |
s |
subject_line |
Units belong in the value, in words: "order_total": "34 dollars", not "order_total": 34.
And the field names should match the words in your questions. If the question says "the
customer", the state should say customer_message, not body, so the model does not have to
work out who wrote what. That habit of naming the subject is the fourth rule on
how to write yes/no questions an LLM answers well.
Anything your code can compute exactly, it should compute before the request:
-
Durations, not dates.
"delivered": "5 days ago"rather than"ordered_at"and"delivered_at"timestamps. On our 999-question test set, questions about dates and durations scored 0.598; questions about facts stated in the text scored 0.954. -
Results, not ingredients.
"items_missing": "1 of 3"rather than two lists to compare. -
Flags that are facts. If your system knows the customer has complained before, send
"previous_complaints": "2", or better, decide the rule part in code and skip the question. -
Comparisons already made, when they are the whole point. If the only thing a question
needs is whether the total is over a limit,
"over_free_shipping_limit": "yes"is a field your code fills in exactly. At that point you may not need a question at all.
The reason is measured on small LLMs and arithmetic in yes/no questions: computing is where a small model is weak and where it leans toward yes. The state is your chance to move that work into code, where it is exact.
A plain string is right when there is one text and no structure: a review, a single email, a
chat message. The README's billing example is just "I was charged twice for the same order."
and 27 input tokens.
An object is right when some facts come from your system and some from a person. Keeping them
in separate fields lets a question point at one ("Does the customer say...") and keeps system
data from being mistaken for something the customer wrote. An array works for a short history
of messages, one object per message with an author field.
Nesting deeper than that rarely helps. A flat object with clear names reads better than a nested one with short keys.
Every token in the state is read before any question is answered. On the reference laptop (Intel Core Ultra 7 255H, 16 threads), with the text read from scratch, that is 26 ms for a request of about 30 tokens and 112 ms for one of about 190. A text asked about again is kept, so the second question on it reads only the question: 22 ms for the same 191-token request. Cutting a noisy 400-token record to the 100 tokens the questions need is the single largest speed-up available to you, larger than any server option. Why latency grows with length is on why LLM latency grows with the length of the text.
The hard limit is 8,192 tokens by default for the state plus each question, counted per question, and 256 KB for the state itself. Records that approach it are long documents, and those have their own design, on yes/no questions about long documents.
Can I send JSON to an LLM instead of text? To jevos, yes: state accepts a string or any
JSON object or array, and questions are asked about it directly.
Does the model read the field names? Treat it as if it does. Names that read as English and match the words in your questions make the state easier to answer from.
Should I send raw timestamps? No. Compute the duration in code and send it as words, such as "5 days ago". Date reasoning is one of the model's weakest kinds of question.
How much does a bigger state cost? On the reference laptop, about 26 ms for 30 tokens and 112 ms for 191 tokens read from scratch. Removing fields no question needs is the cheapest speed-up.
Is there a size limit? 8,192 tokens for the state plus each question, by default, and 256 KB for the state.
See also: LLM policy decisions: put the rule in the question, checking text for personal data with yes/no questions and ask a local LLM a yes/no question.
- The refund and billing examples and their token counts, the 26 ms, 112 ms and 22 ms latencies, and the 8,192-token context: the jev README and our measurements on the reference laptop.
- 0.598 on dates and 0.954 on stated facts: our 999-question test set,
jevos-q4_k_m.
From the notes of jev, whose README example already sends "5 days ago" where a raw record would have carried a date.
- Ask a local LLM a yes/no question and get P(yes)
- Zero-shot text classification with yes/no questions
- LLM policy decisions: put the rule in the question
- LLM as a judge on a CPU
- Why a small LLM says yes when the answer is no
- Small LLMs and arithmetic in yes/no questions
- Our held-out benchmark said 0.855, new questions said 0.757
- jevos vs Jev vs Laya for yes/no decisions
- An open-source alternative to Jev for yes/no decisions
- jevos vs the OpenAI API for yes/no classification
- jevos vs Ollama for yes/no decisions
- jevos vs bart-large-mnli for zero-shot classification
- A yes/no LLM vs a fine-tuned BERT classifier
- jevos vs SetFit: zero-shot vs few-shot classification
- jevos vs Llama Guard for content safety checks
- jev serve vs llama.cpp server for classification
- jevos vs LM Studio: a decision server, not a chat app
- Local vs hosted LLM decisions: latency, cost, privacy
- A yes/no LLM vs a business rules engine
- LLM decisions vs keyword rules and regex
- The fastest AI model for yes/no decisions
- What makes a local LLM fast on a CPU
- Why one forward pass beats generating an answer
- Prefill vs decode: where LLM latency comes from
- Why LLM latency grows with the length of the text
- Why a hosted LLM API cannot answer in 50 ms
- Many questions about one text: why the extra ones are cheap
- CPU or GPU for a small LLM
- Latency budgets: where a 200 ms model fits
- Measuring LLM latency: median, p90 and warm-up
- Q4_K_M vs Q8_0: speed and size for a small model
- Throughput vs latency for a decision server
- What P(yes) means, and what it does not
- LLM calibration explained with yes/no answers
- Expected calibration error (ECE), explained
- Temperature scaling for LLM probabilities
- Platt scaling for a yes/no model
- Reading a reliability diagram
- How to choose a threshold for P(yes)
- Thresholds when a wrong yes costs more than a wrong no
- Human in the loop AI with a review band
- Precision and recall at a P(yes) threshold
- Base rates: why a 0.9 yes can still be wrong often
- Combining yes/no answers with AND, OR and NOT
- Logits, log-odds and P(yes)
- LLM confidence scores: probabilities vs self-reports
- How to write yes/no questions an LLM answers well
- Negation in yes/no questions for an LLM
- One condition per question: splitting compound questions
- Ask whether the text says it at all
- Scores as yes/no thresholds: is it at least high?
- Sending JSON as the text: designing the state
- Why wording changes an LLM's answer, and how to test it
- Mainly about: questions for messages with several topics
- Yes/no questions about tone and emotion
- Asking about intent: what does the writer want?
- Yes/no questions about long documents
- Using an English-only LLM with other languages
- Content moderation with a local LLM
- A Discord moderation bot with a local LLM
- Spam detection with yes/no questions
- Review moderation with a local LLM
- Email triage with a local LLM
- Support ticket routing with yes/no questions
- Urgency detection in customer messages
- Sentiment analysis with yes/no questions
- Intent detection with a local LLM
- Lead qualification with yes/no questions
- Fraud case triage with a local LLM
- Phishing email screening with a local LLM
- Log and alert triage with a local LLM
- Checking text for personal data with yes/no questions
- Prompt injection screening with a small model
- Document classification with a local LLM
- Product categorization with yes/no questions
- Contract clause detection with a local LLM
- Refund request triage with a local LLM
- Detecting cancellation intent in customer messages
- RAG evaluation with yes/no questions
- RAG faithfulness check with a local LLM
- Hallucination detection with a local LLM
- LLM regression tests in CI with yes/no checks
- Rubric design for an LLM judge
- Pairwise comparison with a yes/no judge
- LLM judge bias and how to control it
- Evaluation metrics for yes/no classifiers
- Building a yes/no test set for your own data
- Accuracy by kind of question: why one number hides failures
- Generating test questions with answers computed by code
- Benchmark contamination and truly held-out tests
- An LLM router with yes/no questions
- A model cascade: small model first, large model on doubt
- Semantic routing vs yes/no questions
- Gating AI agent tool calls with yes/no checks
- AI agent guardrails with yes/no questions
- Stop conditions for AI agents
- Logging LLM decisions for audit
- Reducing LLM cost with local yes/no decisions
- Replacing chat LLM calls with yes/no questions
- Structured output vs a probability
- A Python client for local LLM decisions
- Calling a local LLM decision server from JavaScript
- Local LLM yes/no decisions in n8n
- A Slack bot that uses local LLM decisions
- Home Assistant automations with local LLM decisions
- A LangChain tool for local yes/no decisions
- Batch decisions from files with jev decide
- Running LLM yes/no checks in GitHub Actions
- Securing a local LLM server with an API key
- curl examples for a local LLM decision API
- Self-hosted AI for decisions
- A private LLM for text classification
- On-premise LLM for business decisions
- GDPR and automated decision-making with an LLM
- Offline AI for decisions: no network needed
- Edge AI decisions on a CPU
- Run an LLM locally without a GPU
- Small language models explained
- When a small model is enough, and when it is not
- An LLM on a laptop: what it can do in real time
- What is GGUF, for someone deploying a classifier
- GGUF quantization types explained: Q4_K_M, Q8_0 and others
- GGUF vs safetensors
- llama.cpp vs Ollama for a classification service
- llama-cpp-python vs calling llama.cpp through ctypes
- llama.cpp on Windows without compiling
- Running llama.cpp CPU only
- Using llama.cpp prebuilt binaries instead of building