-
Notifications
You must be signed in to change notification settings - Fork 132
jevos vs bart large mnli
bart-large-mnli classifies by asking, for each label, whether the text entails a hypothesis such as "This text is about politics."; jevos classifies by asking, for each label, a yes/no question you write, such as "Is this message mainly about a refund?". Both are zero-shot and both score labels independently when you want them to. The NLI model is smaller (0.4B parameters per its card), with a 1,024-position input; jevos is a 1B model with an 8,192-token context and questions that can carry a rule, not only a topic. For short texts and topic labels the NLI model is the lighter tool. For longer texts, or labels that need a condition spelled out, a question is the more expressive unit.
Conflict of interest, in one line: we build jevos; the facts about bart-large-mnli come from its Hugging Face model card and config, fetched 2026-09-29.
The two methods are closer than they look. A hypothesis is a statement the model checks against the text; a yes/no question is the same check phrased as a question. The difference is in what each model was built to check and how much you can say in the check.
This page is how NLI zero-shot works, what changes with a question per label, context length, the speed class of each, languages and licence, and a way to choose.
The model card describes the method in two steps. The text to classify is posed as the premise,
and a hypothesis is built from each candidate label: for the label "politics", the card's
example is This text is about politics. The model scores entailment, neutral and contradiction
for the pair (the three classes in its config), and the probabilities for entailment and
contradiction are converted to a label probability.
By default the labels compete with each other. With multi_label=True, the card says, each class
is calculated independently, which is the right mode when a text can belong to several labels.
The card's usage is one line of the Transformers pipeline:
from transformers import pipeline
classifier = pipeline("zero-shot-classification", model="facebook/bart-large-mnli")A jevos request for the same job asks one question per label about the same text:
{
"model": "jev-latest",
"state": "Hi, my card was charged but the order page still says payment pending.",
"questions": {
"billing": {"type": "noul", "instructions": "Is this message mainly about a payment, a charge or a refund?"},
"shipping": {"type": "noul", "instructions": "Is this message mainly about delivery or tracking of a parcel?"}
}
}Each comes back as its own noul, which is the multi_label=True behaviour by construction: the
probabilities do not sum to 1, and you pick the label, or several, in code.
Three practical differences follow.
- What a label can say. A hypothesis template is usually one pattern filled with a label name. A question can hold a definition and a rule: "Our policy refunds items reported missing within 30 days of delivery. Should this customer get a refund?" is a README example. That is a decision, not a topic.
- Shared reading. jevos reads the text once for every question in a request; three questions took about 66 ms against 49 ms for one. An NLI model scores each premise and hypothesis pair.
-
Structured input. jevos takes a JSON object as
stateas well as a string, so fields such asdelivered: "5 days ago"can be sent as they are.
The full recipe, with the thresholds that handle "no label fits", is on zero-shot text classification with yes/no questions.
bart-large-mnli's config sets max_position_embeddings to 1,024, the most tokens it can take for
premise and hypothesis together. Longer texts are cut or have to be split. jevos has an 8,192-token
context for the text and all questions of a request. For support tickets and reviews neither
limit matters; for contracts, reports or long email threads, the difference decides whether
you chunk. Chunking strategy is on
yes/no questions about long documents.
We have not benchmarked bart-large-mnli, so there is no head-to-head number here. What can be said is the class of each. The NLI method scores one premise and hypothesis pair per label, so its cost grows with the number of labels times the length of the text. jevos runs on a CPU, one shared reading of the text plus a small cost per question: 26 ms for a request of about 30 tokens and 112 ms for about 190 on an Intel Core Ultra 7 255H with 16 threads.
For a third-party reference point, the SetFit documentation reports bart-large-mnli at about 31 ms per sentence in its own zero-shot example, measured on its setup. That number is theirs, not ours, and says nothing about jevos on the same hardware.
The card describes a checkpoint of bart-large trained on the MultiNLI dataset and does not list other languages; check before using it on non-English text. jevos reads English only. For other languages, neither is the right default, and a multilingual model is the honest answer, as using an English-only LLM with other languages explains.
Both are MIT: the card lists MIT for the model, and jevos' code is MIT.
| If your situation is... | Lean toward |
|---|---|
| short English texts, topic labels, Transformers already in the stack | bart-large-mnli |
| labels that need a definition or a policy in them | jevos |
| texts over about a thousand tokens | jevos |
| a CPU-only service, one native binary to deploy | jevos |
| many labels on very short texts, cost per label matters most | measure both |
Whichever you pick, measure on a hundred of your own labelled cases. Zero-shot accuracy depends heavily on how the labels are worded, for both methods.
What is bart-large-mnli? A 0.4B-parameter checkpoint of bart-large trained on MultiNLI, used for zero-shot classification by turning each label into an entailment hypothesis.
Is a yes/no question the same as an NLI hypothesis? Close. Both check one statement against the text; a question can also carry a definition or a rule.
Which handles longer texts? jevos, with 8,192 tokens of context against bart-large-mnli's 1,024 positions.
Can both assign several labels? Yes: bart-large-mnli with multi_label=True, jevos because
every question is answered independently.
Which is more accurate? We have not compared them on the same set. Test both on your labels.
See also: jevos vs SetFit, a yes/no LLM vs a fine-tuned BERT classifier and mainly about: questions for messages with several topics.
- bart-large-mnli method,
multi_label=True, MultiNLI, 0.4B parameters, MIT licence and pipeline example: the model card, fetched 2026-09-29. -
max_position_embeddingsof 1,024 and the three NLI classes: the model's config.json, fetched 2026-09-29. - The 31 ms per sentence figure: SetFit's zero-shot how-to, their measurement, fetched 2026-09-29.
- jevos latency, the three-question timing and the 8,192-token context: our own measurements and the jev README.
From the notes of jev, a yes/no decision model on a laptop CPU; a question per label is a hypothesis per label with room for a rule.
- Ask a local LLM a yes/no question and get P(yes)
- Zero-shot text classification with yes/no questions
- LLM policy decisions: put the rule in the question
- LLM as a judge on a CPU
- Why a small LLM says yes when the answer is no
- Small LLMs and arithmetic in yes/no questions
- Our held-out benchmark said 0.855, new questions said 0.757
- jevos vs Jev vs Laya for yes/no decisions
- An open-source alternative to Jev for yes/no decisions
- jevos vs the OpenAI API for yes/no classification
- jevos vs Ollama for yes/no decisions
- jevos vs bart-large-mnli for zero-shot classification
- A yes/no LLM vs a fine-tuned BERT classifier
- jevos vs SetFit: zero-shot vs few-shot classification
- jevos vs Llama Guard for content safety checks
- jev serve vs llama.cpp server for classification
- jevos vs LM Studio: a decision server, not a chat app
- Local vs hosted LLM decisions: latency, cost, privacy
- A yes/no LLM vs a business rules engine
- LLM decisions vs keyword rules and regex
- The fastest AI model for yes/no decisions
- What makes a local LLM fast on a CPU
- Why one forward pass beats generating an answer
- Prefill vs decode: where LLM latency comes from
- Why LLM latency grows with the length of the text
- Why a hosted LLM API cannot answer in 50 ms
- Many questions about one text: why the extra ones are cheap
- CPU or GPU for a small LLM
- Latency budgets: where a 200 ms model fits
- Measuring LLM latency: median, p90 and warm-up
- Q4_K_M vs Q8_0: speed and size for a small model
- Throughput vs latency for a decision server
- What P(yes) means, and what it does not
- LLM calibration explained with yes/no answers
- Expected calibration error (ECE), explained
- Temperature scaling for LLM probabilities
- Platt scaling for a yes/no model
- Reading a reliability diagram
- How to choose a threshold for P(yes)
- Thresholds when a wrong yes costs more than a wrong no
- Human in the loop AI with a review band
- Precision and recall at a P(yes) threshold
- Base rates: why a 0.9 yes can still be wrong often
- Combining yes/no answers with AND, OR and NOT
- Logits, log-odds and P(yes)
- LLM confidence scores: probabilities vs self-reports
- How to write yes/no questions an LLM answers well
- Negation in yes/no questions for an LLM
- One condition per question: splitting compound questions
- Ask whether the text says it at all
- Scores as yes/no thresholds: is it at least high?
- Sending JSON as the text: designing the state
- Why wording changes an LLM's answer, and how to test it
- Mainly about: questions for messages with several topics
- Yes/no questions about tone and emotion
- Asking about intent: what does the writer want?
- Yes/no questions about long documents
- Using an English-only LLM with other languages
- Content moderation with a local LLM
- A Discord moderation bot with a local LLM
- Spam detection with yes/no questions
- Review moderation with a local LLM
- Email triage with a local LLM
- Support ticket routing with yes/no questions
- Urgency detection in customer messages
- Sentiment analysis with yes/no questions
- Intent detection with a local LLM
- Lead qualification with yes/no questions
- Fraud case triage with a local LLM
- Phishing email screening with a local LLM
- Log and alert triage with a local LLM
- Checking text for personal data with yes/no questions
- Prompt injection screening with a small model
- Document classification with a local LLM
- Product categorization with yes/no questions
- Contract clause detection with a local LLM
- Refund request triage with a local LLM
- Detecting cancellation intent in customer messages
- RAG evaluation with yes/no questions
- RAG faithfulness check with a local LLM
- Hallucination detection with a local LLM
- LLM regression tests in CI with yes/no checks
- Rubric design for an LLM judge
- Pairwise comparison with a yes/no judge
- LLM judge bias and how to control it
- Evaluation metrics for yes/no classifiers
- Building a yes/no test set for your own data
- Accuracy by kind of question: why one number hides failures
- Generating test questions with answers computed by code
- Benchmark contamination and truly held-out tests
- An LLM router with yes/no questions
- A model cascade: small model first, large model on doubt
- Semantic routing vs yes/no questions
- Gating AI agent tool calls with yes/no checks
- AI agent guardrails with yes/no questions
- Stop conditions for AI agents
- Logging LLM decisions for audit
- Reducing LLM cost with local yes/no decisions
- Replacing chat LLM calls with yes/no questions
- Structured output vs a probability
- A Python client for local LLM decisions
- Calling a local LLM decision server from JavaScript
- Local LLM yes/no decisions in n8n
- A Slack bot that uses local LLM decisions
- Home Assistant automations with local LLM decisions
- A LangChain tool for local yes/no decisions
- Batch decisions from files with jev decide
- Running LLM yes/no checks in GitHub Actions
- Securing a local LLM server with an API key
- curl examples for a local LLM decision API
- Self-hosted AI for decisions
- A private LLM for text classification
- On-premise LLM for business decisions
- GDPR and automated decision-making with an LLM
- Offline AI for decisions: no network needed
- Edge AI decisions on a CPU
- Run an LLM locally without a GPU
- Small language models explained
- When a small model is enough, and when it is not
- An LLM on a laptop: what it can do in real time
- What is GGUF, for someone deploying a classifier
- GGUF quantization types explained: Q4_K_M, Q8_0 and others
- GGUF vs safetensors
- llama.cpp vs Ollama for a classification service
- llama-cpp-python vs calling llama.cpp through ctypes
- llama.cpp on Windows without compiling
- Running llama.cpp CPU only
- Using llama.cpp prebuilt binaries instead of building