Repository navigation
contract clause detection with a local llm
A small local LLM can tell you whether a contract appears to contain a given clause, such as automatic renewal, a limitation of liability or termination for convenience, by asking one yes/no question per clause about each section of the text. Presence is a reading question, and reading is what a small model does best. Whether the clause is enforceable, fair, standard for your industry or acceptable to your company is not a reading question, and it stays with a lawyer. The model's job is to point a reviewer at the right pages faster, on your own machine.
The design choice that matters most is the direction of the error. In clause detection a missed clause is usually worse than a false alarm: a reviewer can dismiss a highlighted paragraph in seconds, but nobody reviews a clause they were told is not there.
This page is the questions, the request for one section, long contracts, what not to ask, how to set thresholds so "not found" means something, and where the legal review sits.
Questions about whether the text says something, one clause per question:
| Clause | Question |
|---|---|
| automatic renewal | Does this section say the agreement renews automatically unless someone gives notice? |
| limitation of liability | Does this section limit or cap the amount one party can be liable for? |
| termination for convenience | Does this section allow a party to end the agreement without giving a reason? |
| exclusivity | Does this section prevent a party from working with competitors or other suppliers? |
| governing law | Does this section say which country's or state's law governs the agreement? |
| assignment | Does this section restrict a party from transferring the agreement to someone else? |
Each question describes what the clause does, not its name. Contracts rarely use the same heading twice, and "Does this section contain a limitation of liability clause?" leans on the label, while "Does this section limit or cap the amount one party can be liable for?" asks about the content. On our 999 hand-written questions, the first jevos answered stated facts right 0.954 of the time and paraphrase questions 0.893 (per-kind numbers for jevos-v4 are not published). Clause detection is mostly those two skills: the clause is stated, in words that differ from yours.
{
"model": "jev-latest",
"state": {
"document": "Master services agreement",
"section": "12. Term. This Agreement starts on the Effective Date and continues for twelve months. It will then extend for further periods of twelve months each unless either party gives written notice at least sixty days before the end of the current period."
},
"questions": {
"auto_renewal": {"type": "noul", "instructions": "Does this section say the agreement renews automatically unless someone gives notice?"},
"liability_cap": {"type": "noul", "instructions": "Does this section limit or cap the amount one party can be liable for?"},
"termination_convenience": {"type": "noul", "instructions": "Does this section allow a party to end the agreement without giving a reason?"}
}
}The section never says "renew". A question about the effect ("renews automatically unless
someone gives notice") is what lets the model match "will then extend for further periods".
Each question returns its own noul, and several questions about one section cost little more
than one, because the section is read once.
Each question's prompt, the text plus that question, holds at most 8,192 tokens by default, and on our reference laptop latency grows with the text: about 28 ms for a short request, 130 ms for a long one read from scratch. A long agreement with schedules will not fit, and even when it fits, one question about a hundred pages is worse than the same question about each section.
Split on the contract's own structure (numbered sections or clauses), ask every clause question about every section, and combine with OR: the contract contains the clause if any section does. Keep the section number with each answer, so a reviewer goes straight to it. The method is on yes/no questions about long documents, and the logic of combining answers in code on combining yes/no answers with AND, OR and NOT.
If the first job is to tell a contract from an invoice or a letter, do that first with document classification with a local LLM.
- Legal judgments. "Is this clause enforceable?", "Is this cap reasonable?", "Is this unusual for a software contract?" These need legal knowledge, jurisdiction and context the text does not contain.
- Numbers and dates. "Is the notice period longer than 30 days?" "Does the cap exceed the annual fees?" Extract the figure and compare in code. On the same test set, the first jevos got number against threshold questions right 0.654 of the time and dates 0.598.
- Absence in one go. "Does this contract lack a liability cap?" is a negated question about a whole document. Ask the positive question per section and conclude absence in code when no section says yes. Why negation needs care is on negation in yes/no questions.
Because a missed clause is the costly mistake, set a low threshold for highlighting a section, well below 0.5, and accept more false highlights. Then check what "not found" means on your own contracts: take a few dozen agreements where a lawyer has already marked the clauses, run the questions, and count how many marked clauses scored below your threshold. That count, not a general accuracy figure, tells you whether "no auto-renewal found" is safe to show. We have not measured jevos on contracts, and the 999-question figures describe kinds of questions, not this task.
The general reasoning on asymmetric thresholds is on thresholds when a wrong yes costs more than a wrong no; here it runs the other way, since the wrong no is the expensive one.
At the end, and always. The model can shorten the path to the relevant paragraphs; it cannot read a contract for you in the sense that matters. OWASP's guidance on overreliance on model output recommends human oversight and fact-checking "especially for critical or sensitive information", and a contract a company is about to sign is that. A sensible pipeline:
- Split the contract into sections and ask the clause questions locally.
- Show the reviewer every section above the highlight threshold, grouped by clause, with the section number.
- The reviewer reads the highlights and the parts of the contract the checklist does not cover.
- The reviewer, not the model, records the conclusion.
Running locally is a real benefit here: draft contracts are confidential, and a model on your own CPU sends them nowhere. It does not change who is responsible for the review.
Can an LLM find clauses in a contract? It can answer whether each section appears to contain a clause you describe, which is a reading task. It should point a reviewer at sections, not replace the review.
Can it tell me whether a clause is enforceable? No. That is legal advice and it stays with a lawyer.
How do I handle a 60-page contract? Split it by section, ask each clause question per section, and combine with OR in code. The text plus each question has to fit in 8,192 tokens.
What threshold should I use? A low one for highlighting, because a missed clause costs more than a false highlight. Check it on contracts a lawyer has already marked.
Is it safe for confidential drafts? The model runs on your machine and sends nothing out. Your own storage and access rules still apply.
See also: human in the loop AI with a review band, ask whether the text says it at all and a private LLM for text classification.
- Our measurements: context and latency from the jev README; accuracy by kind of question from our 999 hand-written questions, measured on the first jevos. No contract measurement exists; none is claimed.
- OWASP GenAI Security Project, LLM09:2025 Misinformation, overreliance and human oversight, fetched 2026-09-29.
- This page is not legal advice.
From the notes of jev, a yes/no decision model that runs on a laptop CPU. The renewal clause in the example never uses the word "renew", which is why every question on this page describes what a clause does, not what it is called.
- Ask a local LLM a yes/no question and get P(yes)
- Zero-shot text classification with yes/no questions
- LLM policy decisions: put the rule in the question
- LLM as a judge on a CPU
- What is a decision model in AI?
- AI decision making with probabilities and thresholds
- Why a small LLM says yes when the answer is no
- Small LLMs and arithmetic in yes/no questions
- Our held-out benchmark said 0.855, new questions said 0.757
- Does an LLM know when a fact is missing?
- Snake AI: a local model vs a cloud API
- jevos vs Jev vs Laya for yes/no decisions
- An open-source alternative to Jev for yes/no decisions
- jevos vs the OpenAI API for yes/no classification
- jevos vs Ollama for yes/no decisions
- jevos vs bart-large-mnli for zero-shot classification
- A yes/no LLM vs a fine-tuned BERT classifier
- jevos vs SetFit: zero-shot vs few-shot classification
- jevos vs Llama Guard for content safety checks
- jev serve vs llama.cpp server for classification
- jevos vs LM Studio: a decision server, not a chat app
- Local vs hosted LLM decisions: latency, cost, privacy
- A yes/no LLM vs a business rules engine
- LLM decisions vs keyword rules and regex
- An open-source alternative to Microsoft-Decision-1
- Decision engine vs decision model
- The fastest AI model for yes/no decisions
- What makes a local LLM fast on a CPU
- Why one forward pass beats generating an answer
- Prefill vs decode: where LLM latency comes from
- Why LLM latency grows with the length of the text
- Why a hosted LLM API cannot answer in 50 ms
- Many questions about one text: why the extra ones are cheap
- CPU or GPU for a small LLM
- Latency budgets: where a 200 ms model fits
- Measuring LLM latency: median, p90 and warm-up
- Q4_K_M vs Q8_0: speed and size for a small model
- Throughput vs latency for a decision server
- OpenVINO for LLM inference on a CPU
- INT8 weight quantization explained
- What P(yes) means, and what it does not
- LLM calibration explained with yes/no answers
- Expected calibration error (ECE), explained
- Temperature scaling for LLM probabilities
- Platt scaling for a yes/no model
- Reading a reliability diagram
- How to choose a threshold for P(yes)
- Thresholds when a wrong yes costs more than a wrong no
- Human in the loop AI with a review band
- Precision and recall at a P(yes) threshold
- Base rates: why a 0.9 yes can still be wrong often
- Combining yes/no answers with AND, OR and NOT
- Logits, log-odds and P(yes)
- LLM confidence scores: probabilities vs self-reports
- How to write yes/no questions an LLM answers well
- Negation in yes/no questions for an LLM
- One condition per question: splitting compound questions
- Ask whether the text says it at all
- Scores as yes/no thresholds: is it at least high?
- Sending JSON as the text: designing the state
- Why wording changes an LLM's answer, and how to test it
- Mainly about: questions for messages with several topics
- Yes/no questions about tone and emotion
- Asking about intent: what does the writer want?
- Yes/no questions about long documents
- Using an English-only LLM with other languages
- Multi-class text classification with a choice question
- Content moderation with a local LLM
- A Discord moderation bot with a local LLM
- Spam detection with yes/no questions
- Review moderation with a local LLM
- Email triage with a local LLM
- Support ticket routing with yes/no questions
- Urgency detection in customer messages
- Sentiment analysis with yes/no questions
- Intent detection with a local LLM
- Lead qualification with yes/no questions
- Fraud case triage with a local LLM
- Phishing email screening with a local LLM
- Log and alert triage with a local LLM
- Checking text for personal data with yes/no questions
- Prompt injection screening with a small model
- Document classification with a local LLM
- Product categorization with yes/no questions
- Contract clause detection with a local LLM
- Refund request triage with a local LLM
- Detecting cancellation intent in customer messages
- Reranking search results with yes/no questions
- RAG evaluation with yes/no questions
- RAG faithfulness check with a local LLM
- Hallucination detection with a local LLM
- LLM regression tests in CI with yes/no checks
- Rubric design for an LLM judge
- Pairwise comparison with a yes/no judge
- LLM judge bias and how to control it
- Evaluation metrics for yes/no classifiers
- Building a yes/no test set for your own data
- Accuracy by kind of question: why one number hides failures
- Generating test questions with answers computed by code
- Benchmark contamination and truly held-out tests
- An LLM router with yes/no questions
- A model cascade: small model first, large model on doubt
- Semantic routing vs yes/no questions
- Gating AI agent tool calls with yes/no checks
- AI agent guardrails with yes/no questions
- Stop conditions for AI agents
- Logging LLM decisions for audit
- Reducing LLM cost with local yes/no decisions
- Replacing chat LLM calls with yes/no questions
- Structured output vs a probability
- A Python client for local LLM decisions
- Calling a local LLM decision server from JavaScript
- Local LLM yes/no decisions in n8n
- A Slack bot that uses local LLM decisions
- Home Assistant automations with local LLM decisions
- A LangChain tool for local yes/no decisions
- Batch decisions from files with jev decide
- Running LLM yes/no checks in GitHub Actions
- Securing a local LLM server with an API key
- curl examples for a local LLM decision API
- Self-hosted AI for decisions
- A private LLM for text classification
- On-premise LLM for business decisions
- GDPR and automated decision-making with an LLM
- Offline AI for decisions: no network needed
- Edge AI decisions on a CPU
- Run an LLM locally without a GPU
- Small language models explained
- When a small model is enough, and when it is not
- An LLM on a laptop: what it can do in real time
- What is GGUF, for someone deploying a classifier
- GGUF quantization types explained: Q4_K_M, Q8_0 and others
- GGUF vs safetensors
- llama.cpp vs Ollama for a classification service
- llama-cpp-python vs calling llama.cpp through ctypes
- llama.cpp on Windows without compiling
- Running llama.cpp CPU only
- Using llama.cpp prebuilt binaries instead of building
- Download a GGUF model from Hugging Face