Repository navigation
int8 weight quantization explained
INT8 weight quantization stores each weight as an 8-bit integer, plus a scale (and sometimes a zero point) shared by a row or block of weights to map the integers back to real values. That is a quarter of the 32-bit size and half of the 16-bit one, and on a CPU, where a language model often waits on memory, fewer bytes usually means faster passes. The accuracy cost of 8 bits is usually small (OpenVINO rates it "Low") but depends on the model and the task, so check it.
The point often missed is that "INT8" names a width, not a method. Symmetric or asymmetric, one scale per channel or per block of 32, weights only or activations too: two "8-bit" builds of one model are not interchangeable without a test.
This page is the mapping with a worked example, the two schemes, the speed, activations, the accuracy evidence, and GGUF.
The standard form comes from Jacob et al. (2017): a real value r is stored as an integer q with r = S(q - Z), where S is a floating-point scale and Z an integer zero point. It makes sure "the real value r=0 be exactly representable", because operators often pad arrays with zeros. With the range set from the weights' own extremes, the loss is rounding: every weight lands within half a step of where it was.
How many weights share one scale is the granularity. NNCF, OpenVINO's compression library, lists both of its INT8 weight modes as "Per-channel" with FP16 scales: one scale per output channel. Its "group-wise" option, offered for 4-bit modes, gives each group (for example 128 weights) its own parameters. llama.cpp's Q8_0 uses blocks of 32.
Symmetric quantization centers the integer range on zero and stores no zero point; asymmetric quantization shifts the range to fit the real minimum and maximum. In NNCF's definitions, symmetric weights use levels -127 to 127 and asymmetric ones 0 to 255, with the rule that "floating-point zero should be within quantization range and strictly mapped into quant".
An illustrative row of five weights from -0.50 to 0.30 (our arithmetic, not a measurement):
| Weight | Symmetric q (S = 0.50/127) | Back to real | Asymmetric q (S = 0.80/255, Z = 159) | Back to real |
|---|---|---|---|---|
| -0.50 | -127 | -0.500 | 0 | -0.499 |
| -0.12 | -30 | -0.118 | 121 | -0.119 |
| 0.00 | 0 | 0.000 | 159 | 0.000 |
| 0.07 | 18 | 0.071 | 181 | 0.069 |
| 0.30 | 76 | 0.299 | 255 | 0.301 |
Symmetric never uses the codes 77 to 127, because nothing in this row is that large and positive. Asymmetric spends all 256 codes on the range that exists, so its step is about a fifth smaller (0.0031 against 0.0039), for the price of one more stored number per channel. On a row balanced around zero, the two come out about the same.
NNCF's default for weight compression is asymmetric, "INT8_ASYM mode", with INT8_SYM available "for symmetric quantization without zero point". llama.cpp's Q8_0 is symmetric: each block holds a 16-bit scale and 32 signed 8-bit values, nothing else.
A dense model reads essentially all its weights on every pass; fewer bytes take less time to stream. OpenVINO's weight compression page says it "improves inference speed by reducing the latency of memory access when computing the operations with weights".
The sizes are arithmetic: a billion weights take about 4 GB at 32 bits, 2 GB at 16 and 1 GB at 8, plus the scales. OpenVINO's table shows the step from 32 to 8 bits on real models: dolly-v2-3b from 10.3 GB to 2.6 GB, Llama-2-7b-chat from 25.1 GB to 6.3 GB.
Time shrinks less than size, because reading a long prompt leans on arithmetic more than on memory, as prefill vs decode explains; OpenVINO's trade-off table rates its INT8 latency gain "Medium". jevos-v4 runs with 8-bit weights through OpenVINO in about 1 GB of memory with the model loaded; the runtime side is on OpenVINO for LLM inference on a CPU.
Weight-only quantization keeps the activations, the values flowing through the model, in floating point. In OpenVINO's words, unlike full model quantization, "where both weights and activations are quantized", weight compression "only targets weights, keeping activations as floating-point numbers", and needs no sample data to calibrate activation ranges.
Activations are the harder half. SmoothQuant (Xiao et al., 2022) starts from "the fact that weights are easy to quantize while activations are not" and reports W8A8 (8-bit weights and activations) up to 1.56 times faster with half the memory and "negligible loss in accuracy". LLM.int8() (Dettmers et al., 2022) handles "emergent outliers" with a mixed-precision decomposition.
A middle path works at run time: an OpenVINO blog post describes dynamic quantization of the
inputs to matrix multiplications, "effective when weights are already quantized into int4 or
int8", in groups whose size "impacts both performance and accuracy". The post covers GPUs and
says CPUs support it with behavior that "may differ slightly". jev has an option of that name, --dynamic-quantization, default 128,
documented at the top of src/main.cpp.
Usually little on the models measured; nothing you can assume for yours. OpenVINO's data-free table, perplexity on Lambada OpenAI (lower is better):
| Model | FP32 | INT8_ASYM | INT4_ASYM, mixed |
|---|---|---|---|
| dolly-v2-3b | 5.01 | 5.07 | 5.28 |
| Llama-2-7b-chat | 3.28 | 3.29 | 3.41 |
NNCF says both INT8 modes "typically preserve model accuracy", and OpenVINO warns that below 1B parameters "low-bit weight compression may result in more accuracy reduction than with larger models".
Being straight about the limit: perplexity measures how well a model predicts general text, not whether a yes/no answer flips. No comparison of jevos at 8 bits against higher precision is published, so we cannot give that cost for jevos. For any quantized build, run a yes/no test set of your own data and check calibration too, as on LLM calibration explained.
Q8_0's 32 values plus one 16-bit scale cost 8 + 16/32 bits per weight, which is why llama.cpp's quantize README lists 8.5008 for its 8B example. A per-channel scale over a channel of 1,000 weights (illustrative) adds only 0.016 bits. Q4_K_M, at 4.8944 bits per weight in the same README, uses super-blocks of 256 weights, decoded on GGUF quantization types explained.
jevos-v4 comes three ways: the 8-bit OpenVINO model that jev runs, and jevos-v4-q8_0.gguf
(942,872,288 bytes) and jevos-v4-q4_k_m.gguf (619,289,312 bytes) for llama.cpp. The first two
are both "8-bit" yet different formats, and no comparison of their answers is published. Choosing
between the GGUF files is on Q4_K_M vs Q8_0: speed and size.
Is INT8 quantization lossless? No: each weight is rounded to one of 255 or 256 levels, and in OpenVINO's table perplexity moves a little.
What is the difference between symmetric and asymmetric INT8? Whether a zero point is stored. Asymmetric fits the real minimum and maximum, which uses the codes better on lopsided ranges.
Does INT8 make inference faster on a CPU? Usually, with fewer bytes to move; measure on your own hardware.
Should I use INT8 or 4-bit? INT8 stays closer to the original at a larger size; in OpenVINO's table the 4-bit modes save more memory at a larger perplexity cost.
See also: what is GGUF, what makes a local LLM fast on a CPU and small language models explained.
- Our own facts: 8-bit weights through OpenVINO, about 1 GB of memory, the
--dynamic-quantizationdefault (jev README); GGUF byte sizes (Hugging Face model page). The five-weight example and the overhead figures are our arithmetic, illustrative. - All fetched 2026-10-12: Jacob et al., arXiv:1712.05877 (read in its HTML version); NNCF weight compression; NNCF quantization schemes; OpenVINO weight compression; OpenVINO dynamic quantization post; SmoothQuant; LLM.int8(); llama.cpp quantize README; ggml-common.h.
From the notes of jev, which ships jevos as one 8-bit OpenVINO model and two GGUF builds, with no published comparison of their answers yet.
- Ask a local LLM a yes/no question and get P(yes)
- Zero-shot text classification with yes/no questions
- LLM policy decisions: put the rule in the question
- LLM as a judge on a CPU
- What is a decision model in AI?
- AI decision making with probabilities and thresholds
- Why a small LLM says yes when the answer is no
- Small LLMs and arithmetic in yes/no questions
- Our held-out benchmark said 0.855, new questions said 0.757
- Does an LLM know when a fact is missing?
- Snake AI: a local model vs a cloud API
- jevos vs Jev vs Laya for yes/no decisions
- An open-source alternative to Jev for yes/no decisions
- jevos vs the OpenAI API for yes/no classification
- jevos vs Ollama for yes/no decisions
- jevos vs bart-large-mnli for zero-shot classification
- A yes/no LLM vs a fine-tuned BERT classifier
- jevos vs SetFit: zero-shot vs few-shot classification
- jevos vs Llama Guard for content safety checks
- jev serve vs llama.cpp server for classification
- jevos vs LM Studio: a decision server, not a chat app
- Local vs hosted LLM decisions: latency, cost, privacy
- A yes/no LLM vs a business rules engine
- LLM decisions vs keyword rules and regex
- An open-source alternative to Microsoft-Decision-1
- Decision engine vs decision model
- The fastest AI model for yes/no decisions
- What makes a local LLM fast on a CPU
- Why one forward pass beats generating an answer
- Prefill vs decode: where LLM latency comes from
- Why LLM latency grows with the length of the text
- Why a hosted LLM API cannot answer in 50 ms
- Many questions about one text: why the extra ones are cheap
- CPU or GPU for a small LLM
- Latency budgets: where a 200 ms model fits
- Measuring LLM latency: median, p90 and warm-up
- Q4_K_M vs Q8_0: speed and size for a small model
- Throughput vs latency for a decision server
- OpenVINO for LLM inference on a CPU
- INT8 weight quantization explained
- What P(yes) means, and what it does not
- LLM calibration explained with yes/no answers
- Expected calibration error (ECE), explained
- Temperature scaling for LLM probabilities
- Platt scaling for a yes/no model
- Reading a reliability diagram
- How to choose a threshold for P(yes)
- Thresholds when a wrong yes costs more than a wrong no
- Human in the loop AI with a review band
- Precision and recall at a P(yes) threshold
- Base rates: why a 0.9 yes can still be wrong often
- Combining yes/no answers with AND, OR and NOT
- Logits, log-odds and P(yes)
- LLM confidence scores: probabilities vs self-reports
- How to write yes/no questions an LLM answers well
- Negation in yes/no questions for an LLM
- One condition per question: splitting compound questions
- Ask whether the text says it at all
- Scores as yes/no thresholds: is it at least high?
- Sending JSON as the text: designing the state
- Why wording changes an LLM's answer, and how to test it
- Mainly about: questions for messages with several topics
- Yes/no questions about tone and emotion
- Asking about intent: what does the writer want?
- Yes/no questions about long documents
- Using an English-only LLM with other languages
- Multi-class text classification with a choice question
- Content moderation with a local LLM
- A Discord moderation bot with a local LLM
- Spam detection with yes/no questions
- Review moderation with a local LLM
- Email triage with a local LLM
- Support ticket routing with yes/no questions
- Urgency detection in customer messages
- Sentiment analysis with yes/no questions
- Intent detection with a local LLM
- Lead qualification with yes/no questions
- Fraud case triage with a local LLM
- Phishing email screening with a local LLM
- Log and alert triage with a local LLM
- Checking text for personal data with yes/no questions
- Prompt injection screening with a small model
- Document classification with a local LLM
- Product categorization with yes/no questions
- Contract clause detection with a local LLM
- Refund request triage with a local LLM
- Detecting cancellation intent in customer messages
- Reranking search results with yes/no questions
- RAG evaluation with yes/no questions
- RAG faithfulness check with a local LLM
- Hallucination detection with a local LLM
- LLM regression tests in CI with yes/no checks
- Rubric design for an LLM judge
- Pairwise comparison with a yes/no judge
- LLM judge bias and how to control it
- Evaluation metrics for yes/no classifiers
- Building a yes/no test set for your own data
- Accuracy by kind of question: why one number hides failures
- Generating test questions with answers computed by code
- Benchmark contamination and truly held-out tests
- An LLM router with yes/no questions
- A model cascade: small model first, large model on doubt
- Semantic routing vs yes/no questions
- Gating AI agent tool calls with yes/no checks
- AI agent guardrails with yes/no questions
- Stop conditions for AI agents
- Logging LLM decisions for audit
- Reducing LLM cost with local yes/no decisions
- Replacing chat LLM calls with yes/no questions
- Structured output vs a probability
- A Python client for local LLM decisions
- Calling a local LLM decision server from JavaScript
- Local LLM yes/no decisions in n8n
- A Slack bot that uses local LLM decisions
- Home Assistant automations with local LLM decisions
- A LangChain tool for local yes/no decisions
- Batch decisions from files with jev decide
- Running LLM yes/no checks in GitHub Actions
- Securing a local LLM server with an API key
- curl examples for a local LLM decision API
- Self-hosted AI for decisions
- A private LLM for text classification
- On-premise LLM for business decisions
- GDPR and automated decision-making with an LLM
- Offline AI for decisions: no network needed
- Edge AI decisions on a CPU
- Run an LLM locally without a GPU
- Small language models explained
- When a small model is enough, and when it is not
- An LLM on a laptop: what it can do in real time
- What is GGUF, for someone deploying a classifier
- GGUF quantization types explained: Q4_K_M, Q8_0 and others
- GGUF vs safetensors
- llama.cpp vs Ollama for a classification service
- llama-cpp-python vs calling llama.cpp through ctypes
- llama.cpp on Windows without compiling
- Running llama.cpp CPU only
- Using llama.cpp prebuilt binaries instead of building
- Download a GGUF model from Hugging Face