Edge-First Small Language Model Compression & Deployment Pipeline
The Edge SLM Optimizer is a modular, pipeline-based framework that chains existing compilation tools (PyTorch, ONNX Runtime, ExecuTorch, llama.cpp) with reproducible physical measurements. It bridges the gap between high-level model compression and low-level hardware performance by treating optimization as a hardware-software co-design problem.
Figure: End-to-end pipeline for hardware-aware SLM optimization
1. Hardware-Algorithmic Telemetry Syncing Standard benchmarking suites capture isolated software statistics like tokens per second. Edge SLM Optimizer introduces a tight hardware-software feedback loop that couples language model generation lifecycle hooks (Prefill/Decode phases) directly with physical I2C sensor tracking via the INA219 shunt and vcgencmd. This delivers a native Watts-per-Token signature metric, exposing the absolute energy and thermal footprints of specific token compilation paths.
2. Automated Quality & Perplexity Guardrails Quantization processes (torchao, AutoGPTQ) frequently introduce localized functional degradation that traditional regression unit tests miss. This pipeline runs automated verification layers against a localized WikiText-2 validation set alongside specialized MMLU subsets immediately post-compression. If a quantization matrix causes perplexity degradation to exceed the <15% strict threshold, the build is automatically flagged before hardware export.
3. Dual Compilation Topologies Rather than coupling configurations to a single backend target, the pipeline implements an automated abstraction tier exporting to both cross-platform CPU topologies via ONNX Runtime Mobile and bare-metal ARM architectures via ExecuTorch’s XNNPACK delegate. This enables rapid A/B performance validation against direct hardware acceleration primitives like Arm KleidiAI and ARM NEON.
4. Memory-Bandwidth Constrained Acceleration Edge execution is fundamentally choked by memory bandwidth rather than raw compute. To bypass this bottleneck during decoding, the runtime layer abstracts an on-device Speculative Decoding execution loop. It leverages a custom-distilled 100M-parameter draft model to fast-forward token acceptance paths through the primary 1B+ parameter target model, achieving up to a 2× generation speedup without model degradation.
5. Commit-Driven Performance Regression Engine The CI/CD layout extends beyond basic syntax and structural verification. Performance and accuracy characteristics are fully integrated into containerized environments. Changes to quantization hyper-parameters or logit-processing configurations map directly to performance variations versioned per commit, turning code integration into an active hardware-efficiency gate.
| Decision | Rationale |
|---|---|
| torchao over bitsandbytes for INT4 | PyTorch-native, torch.compile compatible, SOTA accuracy, native ExecuTorch export |
| ONNX Runtime + XNNPACK over TFLite | Model-agnostic, mature ARM NEON via Arm Kleidi, official Pi support |
| ExecuTorch over PyTorch Mobile | PyTorch 2.x successor, torch.export integration, XNNPACK delegate, official Pi 4/5 support |
| llama.cpp as baseline | De facto CPU LLM standard, mature ARM NEON, speculative decoding research (GitHub #21453) |
| INA219 over USB power meter | Board-level measurement (after regulator), ±1% accuracy, 1mA resolution, true SoC power |
| CPU-only for MVP | Pi 5 has no GPU for LLM inference; GPU only needed for QAT (skip, use PTQ) and vLLM (skip, use llama.cpp) |
# 1. Clone and setup
git clone https://github.com/aragit/edge-slm-optimizer.git
cd edge-slm-optimizer
python -m venv .venv && source .venv/bin/activate
pip install -e .
# 2. Run tests
pytest tests/ -v
# 3. Download a model
python -m models.download gemma-3-1b
# 4. Quantize and export
python -m quantization.ptq_int8 --model gemma-3-1b
python -m export.onnx_exporter --model gemma-3-1b
# 5. Benchmark
python -m benchmarks.baseline --model gemma-3-1b| Metric | Target | Measurement Tool |
|---|---|---|
| Model size (INT4) | < 500 MB | du -sh model file |
| Latency (Pi 5, INT4) | < 100 ms/token | telemetry/profiler.py |
| Power (Pi 5, sustained) | < 5 W | INA219 / vcgencmd |
| Thermal (25°C ambient) | Zero throttle events | telemetry/thermal.py |
| Perplexity degradation | < 15% vs FP32 | quantization/evaluate.py |
| Speculative decode speedup | > 1.5× | runtime/speculative_decode.py |
| CI test coverage | > 80% | pytest-cov |
| CI pass rate | 100% | All 3 jobs green |
edge-slm-optimizer/
├── README.md # Full documentation, badges, benchmark tables
├── pyproject.toml # Modern Python packaging, deps, pytest config
├── .github/workflows/ci.yml # pytest + ruff/mypy + Docker build + benchmark validation
├── docker/
│ ├── Dockerfile.pi5 # ARM64 Debian-based, non-root, healthcheck
│ ├── Dockerfile.laptop # x86_64 Ubuntu, CUDA optional
│ └── docker-compose.yml # Pi 5 simulation stack (no GPU)
├── models/
│ ├── __init__.py
│ ├── download.py # HuggingFace Hub downloader with cache
│ ├── registry.py # Model registry: gemma-3-1b, phi-3-mini-3.8b
│ └── configs/
│ ├── gemma_3_1b.yaml
│ └── phi_3_mini.yaml
├── quantization/
│ ├── __init__.py
│ ├── base.py # Abstract Quantizer protocol
│ ├── ptq_int8.py # Post-Training Quantization: torch.quantization
│ ├── ptq_int4.py # GPTQ / AWQ via auto-gptq / autoawq
│ ├── qat_int8.py # Quantization-Aware Training stub (torchao QAT)
│ ├── torchao_int4.py # torchao Int4WeightOnlyConfig (SOTA 2026)
│ └── evaluate.py # Perplexity, accuracy, MMLU subset, speed
├── export/
│ ├── __init__.py
│ ├── base.py # Abstract Exporter protocol
│ ├── onnx_exporter.py # torch.onnx.export → ONNX with dynamic axes
│ ├── executorch_exporter.py # torch.export → .pte with XNNPACK delegate
│ ├── gguf_exporter.py # llama.cpp conversion for baseline comparison
│ └── validate.py # Graph sanity, op coverage, unsupported op detection
├── runtime/
│ ├── __init__.py
│ ├── base.py # Abstract Runtime protocol
│ ├── pytorch_runtime.py # Baseline: PyTorch eager + torch.compile
│ ├── onnx_runtime.py # ONNX Runtime with CPU EP + XNNPACK EP
│ ├── executorch_runtime.py # ExecuTorch .pte loader with XNNPACK
│ ├── llama_cpp_runtime.py # llama.cpp GGUF with speculative decoding support
│ └── speculative_decode.py # Draft-target speculative decoding wrapper
├── telemetry/
│ ├── __init__.py
│ ├── power_draw.py # INA219 I2C + vcgencmd + psutil power telemetry
│ ├── thermal.py # CPU freq, throttle detection, temperature
│ ├── profiler.py # End-to-end latency: tokenize → generate → decode
│ └── reporter.py # JSON/CSV benchmark report generation
├── benchmarks/
│ ├── __init__.py
│ ├── suite.py # Orchestrates all runtime × quantization combos
│ ├── baseline.py # FP32 reference on laptop
│ └── pi5_runner.py # Remote SSH execution on Pi 5 via paramiko
├── tests/
│ ├── __init__.py
│ ├── conftest.py # Shared fixtures: mock model, temp dirs
│ ├── test_quantization.py # Shape preservation, perplexity floor, size reduction
│ ├── test_export.py # Graph validity, no unsupported ops, dynamic axes
│ ├── test_runtime.py # Output consistency across runtimes (same logits ±ε)
│ ├── test_telemetry.py # Mock INA219, mock thermal data
│ ├── test_speculative_decode.py # Acceptance rate > 50%, speedup > 1.3×
│ └── test_end_to_end.py # Full pipeline: download → quantize → export → benchmark
└── docs/
├── pi5-setup.md # Hardware setup, I2C enable, INA219 wiring
├── quantization-guide.md # PTQ vs QAT, GPTQ vs AWQ, torchao Int4
├── export-guide.md # ONNX vs ExecuTorch vs GGUF trade-offs
└── benchmark-methodology.md # Reproducibility protocol, statistical methods
| Device | Spec | Purpose |
|---|---|---|
| Laptop | x86_64, 16GB RAM, Python 3.12 | Quantization, export, baseline benchmarks |
| Raspberry Pi 5 | 8GB RAM, 64-bit OS, NVMe storage | Edge deployment, power/thermal telemetry |
| INA219 | I2C bus 1, addr 0x40, 0.1Ω shunt | Power measurement (optional, has vcgencmd fallback) |
# All tests (mock data, no hardware required)
pytest tests/ -v --cov
# Specific modules
pytest tests/test_quantization.py -v
pytest tests/test_export.py -v
pytest tests/test_telemetry.py -v
pytest tests/test_runtime.py -v
pytest tests/test_end_to_end.py -v- Edge Deployment of Small Language Models Pablo Prieto, Pablo Abad (Submitted Nov 2025, revised Dec 2025).
A comprehensive comparative analysis of CPU, GPU, and NPU backends evaluating latency scaling and hardware efficiency metrics that inform this pipeline's dual-export architecture. - Fast Inference from Transformers via Speculative Decoding Yaniv Leviathan, Matan Kalman, Yossi Matias (ICML).
The foundational theoretical framework validating the use of lightweight draft models to accelerate memory-bandwidth-bound edge decoding phases.
- Accelerating Small Language Models with Speculative Decoding An in-depth look at implementing non-blocking multi-model pipelines and optimizing token acceptance rates on consumer and edge hardware architectures. Watch the Presentation.
- ExecuTorch: High-Performance On-Device AI Meta AI / PyTorch Edge Architecture — Core documentation on utilizing the XNNPACK delegate to target ARM NEON and vector processing units natively.
- torchao: Architecture Optimization GitHub Repository — Foundational repository for PyTorch-native low-bit quantization, weight-only packing formats, and sparsity techniques utilized in our INT4 pipeline modules.
- Arm KleidiAI: Accelerating AI on ARM CPUs Arm Developer Specifications — Key micro-kernel optimizations targeting modern vector extensions (like ARM NEON and SME2) implemented under the hood by our ONNX Runtime and ExecuTorch runtimes.
@software{edge_slm_optimizer,
title = {Edge SLM Optimizer},
author = {aragit},
year = {2026},
url = {https://github.com/aragit/edge-slm-optimizer}
}MIT — see LICENSE for details.