Skip to content

Inference Software

Jump to bottom
Richard Huang edited this page Aug 3, 2026 · 2 revisions

Inference Software

The Inference Software team owns everything between a trained model and the hardware executing it. The job is to take a model expressed in a training framework and turn it into an efficient, numerically correct execution on the target accelerator — then keep it fast as models, workloads, and hardware change.

Documentation

Objectives

  • Execute models correctly, within an agreed numerical tolerance
  • Maximize throughput and minimize latency on the target hardware
  • Use memory bandwidth and capacity efficiently
  • Keep the stack maintainable as new models and operators appear
  • Give hardware and architecture teams actionable performance feedback

Flow Overview

Trained model (PyTorch / ONNX / SafeTensors)
 ↓
Model Frontend          — import, trace, normalize to an IR
 ↓
Graph Compiler          — fuse, tile, lay out, lower to hardware ops
 ↓
Quantization            — reduce precision, calibrate, verify accuracy
 ↓
Kernel Library          — the actual compute implementations
 ↓
Memory Management       — weights, activations, KV cache allocation
 ↓
Batching and Scheduling — decide what runs when
 ↓
Runtime                 — execute on device, manage state
 ↓
Performance Optimization — measure, find bottleneck, iterate

The flow is a pipeline in structure but a loop in practice: performance findings feed back into compiler decisions, quantization choices, and sometimes the hardware architecture itself.

Why LLM Inference Is Its Own Discipline

Serving a large language model splits into two phases with completely different characteristics:

Phase Processes Bound by Arithmetic intensity
Prefill The whole input prompt at once Compute High — large matrix-matrix work
Decode One token at a time, autoregressively Memory bandwidth Low — matrix-vector work

This split drives nearly every design decision in the stack. Decode reads the entire weight set and the growing KV cache to produce a single token, so it is starved for bandwidth rather than arithmetic. Techniques that help prefill often do nothing for decode, and vice versa.

Most of the difficulty — and most of the opportunity — lives in decode.

Interfaces to Other Teams

Team Interface
Architecture Instruction set, memory hierarchy, what the hardware can express
RTL Behavior of each hardware block the compiler targets
Verification Reference models and numerical parity checks
Physical Design Achievable frequency and power, which set realistic performance targets

See also: Architecture · RTL · Verification · Physical Design

Clone this wiki locally