Carrying my ami model from research checkpoint to product: ONNX export with a parity proof, INT8 quantization with the accuracy re-measured instead of assumed, the full latency story on a 2014 CPU, and a demo where the model runs in your browser with no server anywhere.
Try it: huggingface.co/spaces/kenny0bi/yoruba-diacritics (mirrored at kenny0bi.github.io/gbe)
gbé is Yoruba for "to carry". My ami project ended where most research projects end: a trained checkpoint, an evaluation table, a repo. Almost nobody types Yoruba on a machine with PyTorch installed. This project is the second half of the job: making the same model deployable, and proving nothing was lost on the way.
Export. gbe/export.py pulls the published checkpoint from the Hub (gbe deliberately shares no code with the ami repo; the model snapshot is the single source of truth) and exports it to ONNX with dynamic batch and sequence axes. Then the proof: random batches through both graphs, max absolute logit difference 9.5e-6, and every one of the 6,573 MENYO-20k test sentences restores to the byte-identical string under torch and under ONNX. Export without a parity check is a rumor.
Quantize. gbe/quantize.py applies dynamic INT8 quantization: weights stored as int8 with per-channel scales, activations quantized on the fly. 5.17 MB becomes 1.32 MB (3.92x).
Re-measure. Quantization guides say "minimal accuracy loss". I re-ran ami's entire published evaluation for every backend rather than take anyone's word, mine included:
| backend | char acc | word acc | ambiguous words | sentence exact |
|---|---|---|---|---|
| torch fp32 | 92.38% | 85.97% | 88.30% | 10.62% |
| onnx fp32 | 92.38% | 85.97% | 88.30% | 10.62% |
| onnx int8 | 92.39% | 85.99% | 88.32% | 10.63% |
The int8 model comes out 30 net characters ahead out of 367,099 scored positions: quantization noise flipped a handful of coin-toss argmaxes, slightly more of them right than wrong. The honest reading is a tie, and the honest cost of the 8 bits is zero.
Single stream, batch of one, one thread, real test sentences: torch p50 is 12.6 ms; int8 p50 is 4.4 ms (2.9x), with p90 under torch's median. Cold start is the quiet win: 1.7 s for the torch stack, 100 ms for the int8 session, which is the difference between a serverless function that works and one that times out.
An LSTM cannot parallelise across time, so latency is a clean line in sentence length for every backend; int8 just makes each step 2.7x cheaper (12.0 to 4.4 ms per 100 characters).
One scale per channel, s = max|w|/127, and the dot product factors: y = s_w s_x Σ q p. The expensive loop becomes pure integer arithmetic with two float multiplies at the end, and the worst rounding error per weight, s/2, is far below the noise the weights were trained in. The animation's numbers are the actual values in this repo's exported graph. (Source: assets/manim_eightbits.py, video in assets/eightbits.mp4.)
The endgame of shrinking a model is that there is nothing left to host: 1.3 MB ships to the browser once and runs there through onnxruntime-web, about half the weight of a typical web page. The Space is a static page; your text never leaves your machine, and it works offline after the first load. (Found along the way: Gradio Spaces on the free CPU tier now require a paid plan, static Spaces stay free, which makes client-side inference the free tier's power move.)
The browser port re-implements only the thin orthography layer (strip, encode, legality mask, mark application) in JavaScript, with the vocabulary literal generated from the model's vocab.json rather than typed; I checked the in-browser logits against Python's on the same input and they agree to four decimals.
- Dynamic quantization, not static: activations are quantized per call. For an LSTM on CPU this is the standard choice, but it is not the last word; static calibration might buy a little more speed.
- The 2.9x is measured on my 2014 quad-core i7 at one thread. Ratios will differ on modern CPUs (usually in int8's favour, with VNNI).
- The eval-time deltas above carry ami's inherited caveats: formal-register test set, standard orthography.
- Llama-class GPU serving concerns (KV caches, batching) are out of scope: this is the small-model CPU serving story, on purpose.
python -m venv .venv && .venv/bin/pip install numpy==1.26.4 torch==2.2.2 \
onnx onnxruntime huggingface_hub pytest
bash data/get_data.sh # MENYO-20k for the re-evaluation
.venv/bin/python -m gbe.export # ONNX + parity proof
.venv/bin/python -m gbe.quantize # INT8
.venv/bin/python -m pytest tests/ -q # 7 contracts
.venv/bin/python benchmarks/accuracy.py # the table above (~4 min)
.venv/bin/python benchmarks/latency.py # the latency story
.venv/bin/python assets/make_visuals.py # the four figures- gbe/upstream.py the model snapshot as the single source of truth: weights, vocab, and the orthography modules are imported from the published repo, never copied
- gbe/export.py export + the parity proof
- gbe/quantize.py INT8, with sizes recorded
- gbe/runtime.py the serving runtime: onnxruntime + numpy, torch nowhere in sight; one shared decode so backend comparisons measure only the model swap
- benchmarks/ accuracy re-evaluation and latency suite
- web/ the browser app deployed to the Space and to Pages
- tests/ contracts, including torch/onnx agreement pinned on real sentences