Skip to content

Runtime

Jump to bottom
Richard Huang edited this page Aug 3, 2026 · 1 revision

Runtime

The runtime executes the plan. It owns the device, dispatches work, moves data, tracks per-request state, and turns a compiled model plus a schedule into tokens coming out of an API.

Everything above it is a decision; the runtime is where decisions become execution.

Objectives

  • Dispatch efficiently — per-step overhead must be small relative to the step
  • Keep the device busy — overlap host work, data movement, and compute
  • Manage state correctly — every request's cache and position must stay consistent
  • Handle failure — errors on one request must not corrupt the rest
  • Stay observable — you cannot fix what you cannot see

Where It Fits

flowchart LR
    A[Batching and Scheduling] --> B[Runtime]
    C[Kernel Library] --> B
    D[Memory Management] --> B
    B --> E[Generated tokens]
    B --> A
    style B fill:#2d6a9f,color:#fff
Loading

The Execution Loop

At the center of any LLM runtime is a loop that runs once per decode step:

while requests remain:
    batch      = scheduler.form_batch()      # who runs this step
    inputs     = gather_state(batch)         # tokens, positions, cache handles
    logits     = execute_model(inputs)       # the forward pass
    tokens     = sample(logits, batch)       # per-request sampling
    update_state(batch, tokens)              # append to cache, advance positions
    emit(tokens)                             # stream to clients
    retire_finished(batch)                   # free memory, close streams

Every line has a latency budget. In decode, a step may take only a few milliseconds, so host-side work that would be negligible in a training loop becomes a dominant cost here.

The overhead problem: if the forward pass takes 5 ms and per-step host work takes 3 ms, nearly 40% of capacity is lost to bookkeeping. This is why mature runtimes pre-build dispatch sequences, avoid allocation in the loop, and keep the host off the critical path wherever possible.

Dispatch Strategies

Strategy How it works Overhead Flexibility
Eager Host issues each operation as it goes High Full
Captured graph Record a dispatch sequence once, replay it Very low Fixed shapes only
Command buffers Pre-encode work, submit in batches Low Moderate
On-device control Device sequences its own work Lowest Requires hardware support

Graph capture is the standard technique for decode, because decode steps are structurally identical — same operations, same shapes, only the data changes. Capture once per shape bucket, then replay with negligible host cost.

The catch is that captured graphs require static shapes, which is why Memory Management pages the cache into fixed blocks and why the scheduler works in shape buckets. These design choices are connected.

Asynchrony and Overlap

The device should never wait for the host. Standard techniques:

  • Asynchronous dispatch — the host submits work and continues without blocking
  • Multiple streams / queues — independent work proceeds concurrently
  • Overlapped transfer — copy the next step's inputs while computing the current step
  • Deferred synchronization — only synchronize when a result is genuinely needed

The classic mistake is synchronizing to inspect a value — checking whether a request finished by reading a device tensor forces a stall that serializes the entire pipeline. Sampling and stopping decisions should be structured so this isn't necessary, or batched so it happens once per step at a defined point.

State Management

Each in-flight request carries state the runtime must track:

State Notes
Token history Generated so far; needed for stop conditions and output
Position Current sequence index; drives positional encoding
Cache handle Block table into paged KV storage
Sampling parameters Temperature, penalties, seed — per request
Stop conditions Length limits, stop strings, end-of-sequence
Client stream Where to send tokens as they emerge

Because a batch mixes requests at different positions with different parameters, the runtime must apply per-request sampling within a batched forward pass. Getting this wrong — applying one request's temperature to another — produces output that looks plausible and is completely incorrect, a bug that can survive a long time undetected.

Sampling

The forward pass produces logits; sampling turns them into tokens.

Method Behavior
Greedy Always the highest-probability token; deterministic
Temperature Rescale logits to control randomness
Top-k Restrict to the k most likely tokens
Top-p (nucleus) Restrict to the smallest set exceeding cumulative probability p
Penalties Discourage repetition

Sampling is small in FLOPs but sits directly on the critical path, and it involves sorting or partial selection over the vocabulary — which is large. A poorly implemented sampler can consume a surprising fraction of decode step time.

Determinism is worth designing for. Reproducible output requires a per-request seed, deterministic kernel behavior, and batch-invariant numerics — the last being genuinely hard, since results can change with batch composition. Decide early whether you guarantee it.

Error Handling

The runtime is a multi-tenant system, and isolation matters:

  • One request's failure must not affect others in the batch
  • Device errors must be detected and surfaced, not silently ignored
  • Resources from a failed request must be reclaimed
  • Malformed input should be rejected at admission, not mid-forward-pass

Metrics

Metric Why it matters
Step time The fundamental unit; everything is built on it
Host overhead per step Directly reduces achievable throughput
Device utilization Whether hardware is actually busy
Dispatch count per step Proxy for overhead; fewer is better
Synchronization stalls Reveals accidental serialization
Sampling time share Often larger than expected
Error rate by class Distinguishes user errors from system faults

Common Problems

Problem Root cause Fix
Low utilization, fast kernels Host-side overhead dominating Graph capture; remove allocation from the loop
Periodic latency spikes Garbage collection or allocation Pre-allocate; pool objects
Device stalls between steps Synchronizing to read a result Defer synchronization; batch the check
Wrong output for some requests Per-request parameters misapplied in a batch Index sampling parameters per row; test mixed batches
Memory leak over hours Failed requests not reclaimed Reclaim on every exit path, including errors
Nondeterministic output Batch-dependent numerics Fix seeds; use batch-invariant reductions if required
Sampling dominates the profile Naive full-vocabulary sort Partial selection; fused sampling kernel

Best Practices

  • Measure host overhead separately from device time. They have completely different fixes.
  • Capture the decode step as a graph. It's structurally identical every time.
  • Never allocate in the loop. Pre-allocate and reuse.
  • Never synchronize to inspect. Structure control flow to avoid it.
  • Test heterogeneous batches — mixed sampling parameters, mixed lengths, requests finishing mid-batch. This is where correctness bugs live.
  • Reclaim on every path. Especially error paths.
  • Stream tokens as they're produced. Perceived latency improves dramatically even when total time doesn't change.
  • Instrument the step. A per-step breakdown of dispatch, compute, sample, and update is the most useful diagnostic you can have.

See also: Batching and Scheduling · Memory Management · Kernel Library · Performance Optimization · Inference Software

Clone this wiki locally