-
Notifications
You must be signed in to change notification settings - Fork 132
self hosted ai for decisions
Self-hosting a decision model means running three things you control: a model file, a
runtime that executes it, and a server on a port that your application calls. For jevos that
is the model folder from jevos-v2-openvino-int8.zip, the prebuilt jev binary for your
platform with OpenVINO's libraries beside it, and jev serve listening on 127.0.0.1:8017.
There is no GPU to provision and no account to open. What you take on instead is the
work a hosted API did quietly: knowing which model answered, updating it deliberately, keeping
the port private, and noticing when it is down.
The non-obvious part is that the hard problem of self-hosting a large chat model, finding hardware that can run it, mostly disappears for a small decision model. What remains is ordinary service operations, and the one habit worth building early is identifying the model by its file hash rather than by its name.
This page is what you install, how to run it as a service, how to know which model is answering, how to update it, what stays your job, and when a hosted API is still the better choice.
Three pieces, all from the
release page, with SHA256SUMS.txt
next to them:
- The
jevfolder, fromjev-linux-x64.tar.gz,jev-windows-x64.ziporjev-macos-arm64.tar.gz: the binary, OpenVINO's libraries and the licenses. Nothing is compiled, and there is no Python to install. - The model,
jevos-v2-openvino-int8.zip, unpacked into thejevfolder asjev/model. - The server, started from the
jevfolder (.\jev.exe serveon Windows):
./jev serve--threads defaults to all logical CPUs; set it lower if other heavy programs run on the same
machine. --host and --port change where it listens, and --model-dir points at a model
folder elsewhere. If you
do not need a server at all, jev decide answers one request file and exits, which suits batch
jobs; see batch decisions from files with jev decide.
The server is one process holding one model. Treat it like any other internal HTTP dependency.
-
Readiness.
GET /healthreturns{"status": "ready", ...}once the model is loaded. Point your service manager's health check, or your load balancer, at it, and do not send traffic before it answers. -
Timing. Every successful response carries a
Server-Timingheader with the inference time and the total. Log it next to your own wall-clock measurement, so you can tell the model's cost from your network's. - Capacity. The measured figures come from one laptop, an Intel Core Ultra 7 255H with 16 threads: 26 ms for a short request and 112 ms for a 191-token one read from scratch, and 8.7 requests per second from one client, 10.1 from eight. Your hardware and inputs will differ; measure them as described on measuring LLM latency.
-
Restarts. Loading takes time and memory. A supervisor that restarts the process on crash
and waits for
/healthbefore routing to it is enough for most setups.
By hash. Answers name the served model, jevos-v2, but a name is not an identity. Check the
release archives you unpack against SHA256SUMS.txt, and record those hashes with every
deployment: two servers unpacked from the same archives run the same binary on the same model.
This matters more than it seems. A file renamed on disk, a copy that did not finish, or a different quantization behind the same file name all look identical in a config file and give different answers. Recording the hashes in every decision log turns "which model made this decision in March?" into a lookup; logging LLM decisions for audit shows what else to keep with it.
Nothing updates itself. A new model means a new model folder, and a new runtime means a new
jev release. That is the property you want from a decision service, where silent changes
of behaviour are the expensive kind. A reasonable update routine:
- Download the new file and check it against
SHA256SUMS.txt. - Run your own labelled test set against old and new side by side, per kind of question; a hundred real cases is a useful start, as on building a yes/no test set.
- Re-check your thresholds, because a new model's probabilities are not the old model's.
- Switch, and keep the old file until the logs show the new one behaves.
The runtime works the same way. OpenVINO's libraries ship inside the jev folder, so the
runtime changes only when you unpack a new release, and its hash is part of what you record;
the reasons for pinning a runtime are on
using llama.cpp prebuilt binaries.
Self-hosting moves the model onto your machine and the responsibilities with it.
-
Access. The server listens on
127.0.0.1by default, so only the same machine can reach it. If you bind it to another interface, setJEV_API_KEYwhen you start it: every call except/healththen needsAuthorization: Bearer <key>, and a missing or wrong key gets a 401. The server does not terminate TLS; put a reverse proxy in front if the traffic leaves the host. Details on securing a local LLM server with an API key. - Data. The text never leaves the machine, but your application, your proxy and your logs can still keep it. Self-hosting does not decide retention for you.
- Quality. A hosted provider may improve its model under you; a self-hosted file does not change. That is stability, and it is also the reason to re-measure when your inputs drift.
Being straight about the limit: jevos answers yes/no and multiple-choice questions in English and, early, scores (54% on held-out score questions). On 2,000 yes/no questions about business policies none of the models was tuned on, the hosted Jev was right 0.927 of the time against 0.810 for jevos. If the decision needs that accuracy, other languages, or scores, the hosted model is the right tool, and because the wire format is the same, moving between the two is a base URL change. The broader trade-off is on local vs hosted LLM decisions.
What does self-hosted AI mean? Running the model on hardware you control instead of calling someone else's API. For a decision model that is a file, a runtime and a local server.
Do I need a GPU to self-host? Not for jevos. jev runs on the CPU only: x86-64 with AVX2, or Apple silicon.
How do I update a self-hosted model safely? Verify the new file's hash, test it against your own labelled cases, re-check thresholds, then switch, keeping the old file.
Can other machines call it? Only if you bind it to a reachable interface. Set JEV_API_KEY
first, and add a reverse proxy for TLS.
Is it free? The code is MIT and the model file is a download; the cost is your hardware and your time running it.
See also: offline AI for decisions, on-premise LLM for business decisions and ask a local LLM a yes/no question.
- Commands, options, endpoints and release files: the jev README.
- Latency, throughput and the 2,000-question comparison: our measurements, reported in the README.
From the notes of jev, a decision server that is one binary beside one model folder, both known by the hashes you record.
- Ask a local LLM a yes/no question and get P(yes)
- Zero-shot text classification with yes/no questions
- LLM policy decisions: put the rule in the question
- LLM as a judge on a CPU
- Why a small LLM says yes when the answer is no
- Small LLMs and arithmetic in yes/no questions
- Our held-out benchmark said 0.855, new questions said 0.757
- jevos vs Jev vs Laya for yes/no decisions
- An open-source alternative to Jev for yes/no decisions
- jevos vs the OpenAI API for yes/no classification
- jevos vs Ollama for yes/no decisions
- jevos vs bart-large-mnli for zero-shot classification
- A yes/no LLM vs a fine-tuned BERT classifier
- jevos vs SetFit: zero-shot vs few-shot classification
- jevos vs Llama Guard for content safety checks
- jev serve vs llama.cpp server for classification
- jevos vs LM Studio: a decision server, not a chat app
- Local vs hosted LLM decisions: latency, cost, privacy
- A yes/no LLM vs a business rules engine
- LLM decisions vs keyword rules and regex
- The fastest AI model for yes/no decisions
- What makes a local LLM fast on a CPU
- Why one forward pass beats generating an answer
- Prefill vs decode: where LLM latency comes from
- Why LLM latency grows with the length of the text
- Why a hosted LLM API cannot answer in 50 ms
- Many questions about one text: why the extra ones are cheap
- CPU or GPU for a small LLM
- Latency budgets: where a 200 ms model fits
- Measuring LLM latency: median, p90 and warm-up
- Q4_K_M vs Q8_0: speed and size for a small model
- Throughput vs latency for a decision server
- What P(yes) means, and what it does not
- LLM calibration explained with yes/no answers
- Expected calibration error (ECE), explained
- Temperature scaling for LLM probabilities
- Platt scaling for a yes/no model
- Reading a reliability diagram
- How to choose a threshold for P(yes)
- Thresholds when a wrong yes costs more than a wrong no
- Human in the loop AI with a review band
- Precision and recall at a P(yes) threshold
- Base rates: why a 0.9 yes can still be wrong often
- Combining yes/no answers with AND, OR and NOT
- Logits, log-odds and P(yes)
- LLM confidence scores: probabilities vs self-reports
- How to write yes/no questions an LLM answers well
- Negation in yes/no questions for an LLM
- One condition per question: splitting compound questions
- Ask whether the text says it at all
- Scores as yes/no thresholds: is it at least high?
- Sending JSON as the text: designing the state
- Why wording changes an LLM's answer, and how to test it
- Mainly about: questions for messages with several topics
- Yes/no questions about tone and emotion
- Asking about intent: what does the writer want?
- Yes/no questions about long documents
- Using an English-only LLM with other languages
- Content moderation with a local LLM
- A Discord moderation bot with a local LLM
- Spam detection with yes/no questions
- Review moderation with a local LLM
- Email triage with a local LLM
- Support ticket routing with yes/no questions
- Urgency detection in customer messages
- Sentiment analysis with yes/no questions
- Intent detection with a local LLM
- Lead qualification with yes/no questions
- Fraud case triage with a local LLM
- Phishing email screening with a local LLM
- Log and alert triage with a local LLM
- Checking text for personal data with yes/no questions
- Prompt injection screening with a small model
- Document classification with a local LLM
- Product categorization with yes/no questions
- Contract clause detection with a local LLM
- Refund request triage with a local LLM
- Detecting cancellation intent in customer messages
- RAG evaluation with yes/no questions
- RAG faithfulness check with a local LLM
- Hallucination detection with a local LLM
- LLM regression tests in CI with yes/no checks
- Rubric design for an LLM judge
- Pairwise comparison with a yes/no judge
- LLM judge bias and how to control it
- Evaluation metrics for yes/no classifiers
- Building a yes/no test set for your own data
- Accuracy by kind of question: why one number hides failures
- Generating test questions with answers computed by code
- Benchmark contamination and truly held-out tests
- An LLM router with yes/no questions
- A model cascade: small model first, large model on doubt
- Semantic routing vs yes/no questions
- Gating AI agent tool calls with yes/no checks
- AI agent guardrails with yes/no questions
- Stop conditions for AI agents
- Logging LLM decisions for audit
- Reducing LLM cost with local yes/no decisions
- Replacing chat LLM calls with yes/no questions
- Structured output vs a probability
- A Python client for local LLM decisions
- Calling a local LLM decision server from JavaScript
- Local LLM yes/no decisions in n8n
- A Slack bot that uses local LLM decisions
- Home Assistant automations with local LLM decisions
- A LangChain tool for local yes/no decisions
- Batch decisions from files with jev decide
- Running LLM yes/no checks in GitHub Actions
- Securing a local LLM server with an API key
- curl examples for a local LLM decision API
- Self-hosted AI for decisions
- A private LLM for text classification
- On-premise LLM for business decisions
- GDPR and automated decision-making with an LLM
- Offline AI for decisions: no network needed
- Edge AI decisions on a CPU
- Run an LLM locally without a GPU
- Small language models explained
- When a small model is enough, and when it is not
- An LLM on a laptop: what it can do in real time
- What is GGUF, for someone deploying a classifier
- GGUF quantization types explained: Q4_K_M, Q8_0 and others
- GGUF vs safetensors
- llama.cpp vs Ollama for a classification service
- llama-cpp-python vs calling llama.cpp through ctypes
- llama.cpp on Windows without compiling
- Running llama.cpp CPU only
- Using llama.cpp prebuilt binaries instead of building