Exp 2026 06 13 Quant SIMD Threading

Experiment — Quant/dequant SIMD + OpenMP threading

Date: 2026-06-13 · Status: ⚖️ MIXED — SIMD kept, threading reverted · Path: ggml-ork

Hypothesis

If the activation-quant (F32 → int8) and result-dequant (int32 → F32) are a meaningful share of decode time, then (a) SIMD-friendly loops and (b) OpenMP-parallelizing them should speed up decode.

Implementation

• Commit: 20186ed2 (oRKLLM/llama.cpp-rockchip, ork-backend)
• Symbols: ggml-ork.cpp int8 quant/dequant loops.
• (a) SIMD-friendly: precomputed inverse scale, branchless round (q + copysignf(0.5f, q)), no lrintf. Kept (this is the committed form):

for (int m = 0; m < M; m++) {
    const float * yr = y + (size_t) m*K; int8_t * ar = ai + (size_t) m*K;
    float mx = 1e-9f;
    for (int k = 0; k < K; k++) { float v = fabsf(yr[k]); mx = v > mx ? v : mx; }
    as[m] = mx / 127.0f;
    const float inv = 127.0f / mx;                  // precomputed: multiply, don't divide in the hot loop
    for (int k = 0; k < K; k++) {
        float q = yr[k] * inv;
        int qi = (int) (q + copysignf(0.5f, q));    // branchless round, no lrintf
        ar[k] = (int8_t) (qi > 127 ? 127 : qi < -127 ? -127 : qi);
    }
}

• (b) OpenMP threading of those loops. Reverted (never committed — dropped within 20186ed2). The form that net-hurt:

#pragma omp parallel for                            // ~400 tiny parallel regions/token
for (int m = 0; m < M; m++) { /* per-row quant as above */ }

Results

• SIMD-friendly: 9.66 → 9.78 tok/s (marginal, kept).
• OpenMP threading: 9.66 → 5.69 tok/s — net hurt (~400 tiny parallel regions/token of fork/join overhead + contention with ggml's own threadpool).

Benchmark (standardized — 2026-06-13)

The threaded arm was reverted and never committed, so it can't be re-run to standard. What the standardized profile does show on the reference model (Qwen3-1.7B-w8a8, all-NPU, ORK_PROFILE=1) is that quant/dequant is a tiny slice — so no quant micro-optimization (SIMD or threading) can move decode much:

ork profile: 9653 matmuls | quant 48ms (1%) run 3658ms (98%) dequant 21ms (1%) | 386.1 us/matmul

Quant 1% + dequant 1% = 2% of decode; the NPU submit is 98%. The historical SIMD/threading numbers (9.66 → 9.78 kept; 9.66 → 5.69 threading, reverted) were on Qwen2.5-0.5B via llama-cli.

rkllm comparison: librkllmrt's decode (11.71 tok/s on the same model) is bounded by the same NPU submit, not by quant — so this CPU-side lever is irrelevant to both runtimes. No amount of quant tuning closes the gap to the closed runtime.

Verdict

⚖️ SIMD kept; threading reverted. Definitive conclusion: the ORK path is not quant/dequant-bound — it is NPU-submit-latency-bound (~0.2–0.5 ms × ~200 matmuls/token ≈ the flat ~10 tok/s seen for both 135M and 0.5B). CPU-side micro-optimization is the wrong lever; this experiment is what established that. See Submit-latency-floor profiling.