A model inference endpoint is slow

1. Symptoms
2. Common causes
3. Add timers around each stage
4. Fixes
5. Metrics

Symptoms

An API endpoint receives input, runs a model, returns output, and response time is bad.

Break into stages

• request receive
• input validation
• preprocessing
• CPU -> GPU transfer
• model inference
• GPU -> CPU transfer
• postprocessing
• response serialization
• network response

Add timers around each stage.

import time
import torch

def now():
    return time.perf_counter()

def predict(request):
    t0 = now()

    # parse input
    x = parse_request(request)
    t1 = now()

    # preprocess
    x = preprocess(x)
    t2 = now()

    # move to GPU
    x = x.to("cuda")
    t3 = now()

    # model inference
    torch.cuda.synchronize()
    t4 = now()

    with torch.inference_mode():
        y = model(x)

    torch.cuda.synchronize()
    t5 = now()

    # postprocess
    result = postprocess(y)
    t6 = now()

    print({ "parse": t1 - t0, "preprocess": t2 - t1, "to_gpu": t3 - t2, "model": t5 - t4, "postprocess": t6 - t5, "total": t6 - t0, })

    return result

Common causes

• The model is loaded on every request.
• The model is not on GPU.
• The input is moved CPU -> GPU repeatedly in a bad way.
• The endpoint processes one item at a time instead of batching.
• The code uses training mode instead of inference mode.
• Preprocessing is slow.
• Postprocessing is slow.
• Serialization is slow.
• The first request is slow because of warmup.

Fixes

• Make sure model is loaded once, not per request:

model = load_model()
model.to("cuda")
model.eval()

• Use inference mode:

with torch.inference_mode():
    output = model(input)

• Warm up the model:

with torch.inference_mode():
    for _ in range(3):
        _ = model(dummy_input)

• Avoid per-request heavyweight setup.

Bad:

def predict(x):
    model = load_model()
    model.to("cuda")
    return model(x)

Good:

model = load_model().to("cuda")
model.eval()

def predict(x):
    with torch.inference_mode():
        return model(x)

Metrics

• Baseline p50 latency: X ms
• Baseline p95 (95% of data falls at or below) latency: Y ms

Breakdown:

• preprocessing: A ms
• model inference: B ms
• postprocessing: C ms