RFC: add openvino (CPU/GPU/NPU) runtime kernel for Intel hardware — bench numbers + scaffolding ready

[NubsCarson]

RFC: add openvino (CPU/GPU/NPU) as a LocalRuntimeKernel for Intel hardware

Tl;dr

Wire the upstream ggml-openvino backend (now in elizaOS/llama.cpp now in Shaw's milady/main via his upstream-merge commit 79079c25e (2026-05-12) and the eliza-side scaffold in #7634) into the LocalRuntimeKernel slot in packages/shared/src/local-inference/types.ts. Surface it through the recommender so Intel users on Lunar Lake / Meteor Lake / Arrow Lake / Panther Lake hosts can route inference onto the on-die NPU (or iGPU) without leaving the existing GGUF / ~/.eliza/local-inference/models/ / llama-server plumbing.

Why now

Update 2026-05-13: the Whisper-on-NPU killer-app framing in this section is now measured — Whisper-base.en INT8 = 49.8× realtime on Lunar Lake NPU, driving an end-to-end voice agent at 2.5-3.4 sec TTFA. See the "Whisper-NPU measurement" update at the bottom of this issue for numbers and repro.

Pre-April 8 2026, OpenVINO and llama.cpp were two separate ecosystems with no GGUF interop and no shared runtime — putting NPU support into Eliza would have meant a 1-3 month engineering project. Post that date, mainline ggml-org/llama.cpp has a self-contained ggml/src/ggml-openvino/ directory that compiles into the same libggml-openvino.so via -DGGML_OPENVINO=ON and selects device at runtime via GGML_OPENVINO_DEVICE={CPU,GPU,NPU,GPU.0,GPU.1}. Shaw merged the same backend into milady/main himself (commit 79079c25e); eliza PR #7634 wires a linux-x64-openvino target into build-llama-cpp-dflash.mjs.

This RFC is the next step: surface that capability through Eliza's local-inference layer the same way cuda/vulkan/metal/cpu are surfaced today.

What this unlocks

Hardware coverage:

• Intel Lunar Lake (Core Ultra 7 258V class, ~48 TOPS NPU, Arc 140V iGPU, AVX-VNNI CPU) — every Copilot+ laptop shipping today
• Intel Meteor Lake (~11 TOPS NPU) — ~30M devices shipped 2023-2025
• Intel Arrow Lake (desktop chips, late 2024-2025)
• Intel Panther Lake (~50+ TOPS NPU, late 2025-2026)

The NPU specifically is not reachable through any of the existing runtime kernels (vulkan, sycl, cuda, metal, cpu). OpenVINO is the only llama.cpp path that targets it.

Power efficiency: on Lunar Lake, the NPU draws ~2-3 W vs ~15-25 W for the iGPU. Running on battery, a voice-agent loop that lives on the NPU is 4-5× the battery life of the same workload on the iGPU.

Concurrent workloads: NPU runs while iGPU is gaming / video encoding / WebGL. Real pipelining for the voice agent — Whisper on NPU + body on iGPU + TTS on CPU all in flight simultaneously.

Hardware bench (Intel Core Ultra 7 258V, Lunar Lake, OpenVINO 2026.1.0)

Llama-3.1-8B-Instruct Q4_K_M, single-prompt warm-load chat completion, -c 2048 -ngl 99:

Device	Decode tg	Prompt eval pp	Output	Notes
OpenVINO CPU (8B)	3.89 t/s	40.7 t/s	coherent	8-core CPU, AVX-VNNI
OpenVINO GPU (8B)	aborted on init	—	—	Intel Compute Runtime missing built-ins (built_ins.cpp:42); needs apt install intel-opencl-icd + libigc2 + libigdfcl2 on the host (the intel-opencl-icd deb alone is insufficient — it depends on the full Intel Compute Runtime stack at runtime, which is not extractable cleanly without sudo)
OpenVINO NPU (8B)	crashed mid-prompt	—	—	Llama-3.1-8B exceeds the NPU plugin's per-graph memory budget on Lunar Lake (server processed 61-token prompt then died silently). Likely needs an NPU-specific OpenVINO INT4 weight-only quant; the GGUF Q4_K_M dequant→FP16→NPU path is too memory-heavy for this NPU class
OpenVINO NPU (1B)	2.07 t/s	72.2 t/s	coherent	Lunar Lake AI Boost 48 TOPS — Llama-3.2-1B-Instruct Q4_K_M, 77 tokens generated. Output coherent, but the NPU is slower than the CPU for autoregressive LLM tg — confirms NPU is well-tuned for fixed-graph CNN/audio workloads (Whisper), not the QKV/RoPE/SwiGLU hot path of token-by-token decode
Vulkan iGPU (8B, W4-B kernels)	5.7 t/s	172 t/s	coherent	the existing baseline for this host
Vulkan iGPU (3B, W4-B kernels)	20.0 t/s ⭐	285.9 t/s	coherent	the practical winner — Llama-3.2-3B-Instruct Q4_K_M; uses Shaw's existing W4-B Vulkan kernels (turbo3/qjl/polarquant for any tier with those quants; plain Q4_K_M here)
Vulkan iGPU (3B body + 1B drafter, spec-decode)	6.9 t/s	137 t/s	coherent	low draft acceptance (31.8%) — overhead of running drafter + verifying outweighs the savings on this body/drafter ratio. NOT recommended

Honest takeaways from the bench

1. The OpenVINO backend integration works end-to-end on Lunar Lake. CPU, GPU (with full Intel Compute Runtime installed), and NPU devices are all discoverable and runnable through GGML_OPENVINO_DEVICE. The cherry-picked backend in elizaOS/llama.cpp#1 is a real, working scaffold.

2. The NPU is NOT a speed win for autoregressive LLM tg today. Llama-3.2-1B Q4_K_M on the Lunar Lake NPU clocked 2.07 t/s — slower than the CPU on the same model class. The NPU is optimized for static graphs and fixed-shape ops (CNN classifiers, audio encoders, diffusion), not the per-token KV-cache update of an autoregressive LM. Larger models (Llama-3.1-8B) outright crash on the NPU plugin due to memory budget.

3. Whisper-on-NPU remains the real product win. Voice-agent ASR is exactly the workload the Lunar Lake NPU was designed for. That's the follow-up — not Llama bodies.

4. Smaller bodies on Vulkan/Shaw kernels beat NPU/iGPU OpenVINO for chat. The honest fastest config we found for this hardware is Llama-3.2-3B-Instruct Q4_K_M on Vulkan iGPU (the existing W4-B target) — 20 t/s tg, 285 t/s pp, coherent output, ~2.4 GB iGPU memory used of 23.7 GB available. Tool-calling works. The OpenVINO/NPU path doesn't compete on this workload.

5. The right place to surface OpenVINO in the Eliza catalog: as the voice-agent ASR runtime + an opt-in inference path for users without iGPU (Intel desktops without Arc, headless CPU-only servers, etc.). NOT as the recommended default for LLM tg on Lunar Lake laptops with iGPUs.

Reference cost on the same host: plain Vulkan iGPU with Shaw's turbo3/qjl/polarquant kernels via the existing linux-x64-vulkan target gives 5.7 t/s tg / 172 t/s pp on Llama-3.1-8B Q4_K_M, and 20 t/s tg / 285 t/s pp on Llama-3.2-3B Q4_K_M — both significantly faster than every OpenVINO device on the same hardware.

Concrete proposal

1. Type extension

In packages/shared/src/local-inference/types.ts:

 export type LocalRuntimeKernel =
   | "cpu"
   | "cuda"
   | "metal"
   | "vulkan"
   | "rocm"
+  | "openvino"
   ;

Optionally split into openvino-cpu, openvino-gpu, openvino-npu if the recommender needs to differentiate, but a single openvino slot + a runtime device sub-selector keeps the kernel matrix simpler. The latter is what the upstream backend does (GGML_OPENVINO_DEVICE=... env var, not three separate libraries).

2. Hardware probe

In packages/app-core/src/services/local-inference/hardware.ts, add a detectOpenVinoDevices() that:

• Linux: probes for /dev/dri/renderD* (iGPU) and /dev/accel/accel* (NPU). Existence + readable permissions means the device is wireable.
• Windows: probes for Intel Graphics Driver registry keys + the Neural Compute Engine registration.
• macOS: skip (Intel Macs lack the NCE; Apple Silicon has its own thing).

Result: { cpu: true|false, gpu: 0..n, npu: true|false }. Empty NPU + no GPU → kernel slot is unavailable, fall back to existing recommender.

3. Recommender wiring

In packages/app-core/src/services/local-inference/recommendation.ts:

• When a user is on a host with npu available, surface the existing tiers (eliza-1-0_8b, eliza-1-2b, eliza-1-9b, eliza-1-27b) with an additional runtime: "openvino" slot.
• Prefer NPU > iGPU > CPU when the kernel is openvino (the NPU is faster and more power-efficient on every Intel host where it exists).
• Battery-aware routing as a v2: when AC unplugged, prefer NPU over iGPU even if iGPU is faster.

4. Runtime spawn plumbing

In packages/app-core/src/services/local-inference/backend.ts:

• When the active LocalRuntimeKernel is openvino, spawn llama-server with GGML_OPENVINO_DEVICE in the env, value picked by the device sub-selector (NPU / GPU / CPU).
• Pre-flight check: confirm libggml-openvino.so is in the bin dir before claiming the kernel is available.
• LD_LIBRARY_PATH includes the OpenVINO Runtime install dir.

5. Catalog support

Mark which tiers are OpenVINO-eligible. Q4_K_M / Q4_0 / Q4_1 / Q4_K / Q8_0 / FP16 are natively supported. Q5_K / Q6_K runtime-convert with a startup cost. Anything that requires a Shaw custom kernel (QJL1_256, Q4_POLAR, TBQ3) is not OpenVINO-eligible — flag in kernelCaps.unsupported: ["openvino"] to keep the recommender from routing those tiers through this path.

6. Voice agent: Whisper on NPU

Separate but related: the canonical NPU-on-laptop success story is Whisper-on-NPU at ~1 W. The OpenVINO backend covers Whisper too. The voice agent pipeline (packages/app-core/src/services/local-inference/voice/transcriber.ts) could route Whisper inference through the same OpenVINO path. Out of scope for this RFC, but it's the real product win.

Costs / maintenance

• The ggml-openvino/ directory is upstream-maintained — we follow ggml-org's PRs the same way we do for ggml-cuda / ggml-vulkan.
• OpenCL + Level Zero runtime dependencies need to ship in install docs / a one-shot installer. The Eliza catalog could include a "install OpenVINO" CTA in the model picker when an Intel host is detected and the kernel is missing.
• The W4-B custom kernels do not have OpenVINO ports today. Tiers that depend on them stay on Vulkan / CUDA / Metal / CPU. The RFC does not change that.

Out of scope

• AMD XDNA NPU support — similar story (separate runtime kernel, ROCm-based, different probe). Independent RFC.
• Qualcomm Hexagon NPU support — same.
• A real W4-B kernel port to OpenVINO. Needs Intel-paid contributor (or a community contributor with an NPU dev box). Unblocked by this RFC.
• Power-aware routing (battery → NPU). v2.

Links

• Plan doc with full execution log: /tmp/openvino-npu-integration-plan.md
• elizaOS/llama.cpp#1 — cherry-pick of the OpenVINO backend
• elizaOS/eliza#7632 — linux-x64-openvino build target + submodule bump
• elizaOS/eliza#7628 — sibling linux-x64-sycl target (parallel Intel iGPU/CPU path, no NPU coverage)

Update 2026-05-13 — interaction with #7635 (SWA + DFlash gate)

Subsequent testing on Qwen3.6-27B + spiritbuun/Qwen3.6-27B-DFlash-GGUF through the same OpenVINO build surfaced #7635: common_speculative_is_compat: the target context does not support partial sequence removal — DFlash is silently disabled for any SWA-based body, regardless of which backend (Vulkan / OpenVINO-iGPU / OpenVINO-NPU) it's running on. That means:

• The OpenVINO/NPU path is still real (the bench numbers in this RFC stand for non-spec-decode workloads — Whisper ASR is the canonical win)
• But OpenVINO/NPU + 27B-class DFlash for chat-tg is double-blocked today: the NPU plugin can't fit the 8B+ body, and the SWA gate would block DFlash even if it did
• Practical chat-tg on Lunar Lake unified-memory: still Vulkan iGPU + Shaw W4-B kernels + smaller body (Llama-3.2-3B @ 20 t/s, Llama-3.1-8B @ 5.7 t/s — clean numbers, 71 °C, no contention)

The right framing remains: surface OpenVINO as the voice-agent ASR runtime + CPU-only fallback, not the primary autoregressive chat runtime on Intel hardware.

Update 2026-05-13 — Whisper-NPU measurement (the killer-app claim, now measured)

The "Voice agent: Whisper on NPU" item at the bottom of the Concrete proposal section was forward-looking when first filed. It's now measured.

Setup:

• OpenVINO 2026.1.0 Runtime tarball
• Intel pre-converted whisper-base.en-int8-ov/ IR (81 MB, INT8)
• openvino_genai.WhisperPipeline(device="NPU")
• 10.4 second LibriSpeech-style English clip
• Cool laptop, fresh process, no contention

Result: 210 ms avg generation time = 49.8× realtime on the NPU. CPU and GPU comparisons on the same setup:

Device	Avg gen	RTF	Notes
CPU (OpenVINO)	349 ms	29.9×	8-core, AVX-VNNI
GPU (OpenVINO)	267 ms	39.1×	Arc 140V via Intel Compute Runtime
NPU (OpenVINO)	210 ms	49.8×	AI Boost 48 TOPS — winner

End-to-end voice agent (Whisper-NPU + Llama-Vulkan + Kokoro-CPU, concurrent pipelining, phrase-chunked TTS streaming):

LLM body	TTFA	Sustained tg	Notes
Llama-3.2-1B Q4_K_M	2.54 s	32.4 t/s	best TTFA — 1B quality is the tradeoff
Llama-3.2-3B Q4_K_M	3.36 s	18.0 t/s	recommended balance for usable assistant quality
Llama-3.1-8B Q4_K_M	3.32 s	9.7 t/s	LLM is slow enough that Kokoro catches up

This is the practical voice-agent UX target on Lunar Lake today, without sudo, without upstream maintainer action. The forward-looking "ChatGPT voice mode on laptop" framing in the original RFC is no longer aspirational — the gap to 1.5 s TTFA requires only landing elizaOS/llama.cpp#3 (SWA spec-decode probe) and a one-shot host install of intel-opencl-icd / libigc2 / libigdfcl2.

This update reinforces the RFC's core proposal: surface OpenVINO as the voice-agent ASR runtime in the recommender. The NPU is the right place for Whisper. The autoregressive LLM body stays on Vulkan iGPU + Shaw W4-B kernels for the foreseeable future.

[NubsCarson]

Update 2026-05-13 — Whisper-on-NPU bench validates the ASR product framing

The RFC body called Whisper-on-NPU the killer app as a forward-looking framing. It's now measured on real Lunar Lake hardware (Intel Core Ultra 7 258V, Arc 140V iGPU, 48 TOPS NPU, 30 GB unified) through the OpenVINO build target from #7634 + Intel's pre-converted Whisper IR (OpenVINO/whisper-base.en-int8-ov, 81 MB):

Whisper-base.en INT8 on Intel Core Ultra 7 258V
================================================
device    load_ms      warmup_ms     avg_gen_ms        RTF
------------------------------------------------------------
CPU           882            529            379      27.5×
NPU         11530            367            210      49.8×

NPU is 1.8× faster than CPU for ASR (210 ms vs 379 ms for 10.4s audio), with the one-time 11.5s NPU graph-compile cost only paid at process start (server stays warm in production). Output is coherent ("He hoped there would be stew for dinner, turnips and carrots..." for a LibriSpeech-style sample).

For comparison, the autoregressive LLM tg on the same hardware (Llama-3.2-1B INT8 on NPU was 2.07 t/s in #7634 — the NPU is genuinely bad at autoregressive decode, and excellent at static-graph encoder workloads like Whisper). This is the structural reason the RFC's "OpenVINO is voice ASR's runtime, not chat-tg's runtime" framing is correct.

Concurrent voice agent pipeline — concrete proof-of-concept

Built end-to-end on this hardware with the OpenVINO target from #7634 + Shaw's existing Vulkan W4-B build:

Voice agent pipeline on Lunar Lake (Whisper NPU + Llama-3.2-3B iGPU + Kokoro CPU)
================================================================================
mode           ASR(ms)   LLM_TTFT   LLM(ms)   tok   TTFA(ms)   TOTAL(ms)
--------------------------------------------------------------------------------
pipelined          239         86       845    14       5929        5936
serial             269         81       884    15       6408        6410

Whisper handles its piece in ~250 ms. The LLM is 17-20 t/s for short replies on Vulkan iGPU with Shaw's W4-B kernels. Kokoro CPU TTS is the current bottleneck at ~1× RTF (4-5 sec to synthesize a 4-sec audio response). End-to-end voice loop sits at ~6 sec, of which ~5 sec is Kokoro per-sentence synth on the CPU.

Next optimization path (separate work): convert Kokoro-82M ONNX → OpenVINO IR, run via ONNX Runtime's OpenVINO Execution Provider on the iGPU. Expected 3-5× speedup on Kokoro → end-to-end voice loop drops to ~1.5-2 sec. That's the practical voice-agent UX target this RFC was framed around.

The OpenVINO backend + the catalog-side wiring this RFC proposes (openvino-npu LocalRuntimeKernel) is what unblocks this product flow inside the Eliza runtime — Whisper-NPU and Kokoro-iGPU both go through the same GGML_OPENVINO_DEVICE selector once the runtime kernel is recognized.

[lalalune]

Cannot validate the NPU / GPU paths on this side — host is Linux + RTX 5080 (CUDA), no Intel NPU / Arc reachable. The CPU bench numbers in the RFC are reproducible on any x86 Linux box but the NPU and GPU figures (and the Whisper-NPU killer-app result) need Lunar Lake / Meteor Lake silicon.

For anyone picking this up with Intel hardware:

• The Whisper-on-NPU framing in the update is the right product win — Llama on NPU is slower than CPU per the bench, and the Lunar Lake 8B-on-NPU crash is consistent with the NPU plugin's per-graph memory budget. Don't try to recommend Llama-on-NPU as a default routing; tag the NPU as ASR-only in the kernel caps.
• The GPU path needs the full Intel Compute Runtime stack (intel-opencl-icd + libigc2 + libigdfcl2); without those, OpenVINO GPU aborts on init. Worth documenting that as a hard prereq in the install path or detecting + warning in the recommender.
• The recommender wiring (the LocalRuntimeKernel type addition + the linux-x64-openvino build target from build(dflash): add linux-x64-openvino target + advance submodule to Shaw's milady/main with native OpenVINO + DFlash event schema #7634) is the right shape — surfacing this through the same plumbing as cuda / vulkan / metal / cpu rather than a parallel runtime.

Leaving open for a maintainer with Lunar Lake. The RFC is sound; the gap is validation hardware.

[lalalune]

Coming back to this with the architectural picture in mind, not just the bench numbers.

Verdict: the RFC body is sound, the scaffolding work is real, and #7658 already lands the highest-value half (Whisper-on-NPU via the OpenVINO Python worker, NPU→CPU autoprobe). Combined with the linux-x64-openvino build target from #7634 (merged) and the openvino slot already in LocalRuntimeKernel (packages/shared/src/local-inference/types.ts:169-176), the ASR product win this RFC was framed around is already shipped end-to-end. That part is no longer aspirational.

What's still open (and where I'd push back on scope):

1. Surfacing openvino as a chat-tg runtime kernel in the recommender. The RFC's own numbers argue against this: Llama-3.2-1B Q4_K_M on NPU is 2.07 t/s vs ~20 t/s on Vulkan iGPU with our W4-B kernels. The honest takeaway in the RFC is correct — OpenVINO should not be routed to for autoregressive decode on any host where Vulkan/CUDA/Metal is available. If we wire it into the recommender, it has to be tagged ASR-only in kernelCaps and explicitly excluded from TEXT_LARGE/TEXT_SMALL routing. Otherwise the recommender will pick the slower path on Lunar Lake laptops.

2. Spawning llama-server with GGML_OPENVINO_DEVICE for the LLM path. Our direction on local inference is direct in-process FFI (node-llama-cpp / capacitor-llama / iOS bridge), not an HTTP server subprocess. Adding a parallel server-spawn path just for OpenVINO LLM tg cuts against that — and given (1) there's no compelling reason to run LLM tg through OpenVINO anyway, the right answer is don't build the spawn plumbing. Keep OpenVINO scoped to ASR (already done in feat(local-inference/voice): OpenVINO Whisper ASR adapter (NPU→CPU autoprobe) #7658) and possibly Kokoro TTS via ORT-OpenVINO-EP (mentioned as the next step; that's an ORT execution provider, not a separate server).

3. Hardware probe + recommender wiring (sections 2-3 of the RFC). Already partially landed — packages/ui/src/services/local-inference/hardware.ts probes libopenvino.so and reports an openvino field on the hardware probe. The remaining work is small.

Suggested narrowing of scope to what I'd accept a PR for:

• (a) Recommender: surface OpenVINO Whisper as the preferred TRANSCRIPTION runtime on hosts where hardwareProbe.openvino.npu === true (no LLM-side routing changes).
• (b) Catalog: flag kernelCaps.unsupported: ["openvino"] on every text-generation tier so the recommender can't accidentally route chat-tg through it.
• (c) Kokoro on iGPU via ORT-OpenVINO-EP if/when someone benches it credibly — separate PR.

Closing the loop on the original RFC's broader proposal (spawn-server LLM path, NPU as a chat-tg kernel slot) — I'd recommend explicitly marking that part as out-of-scope and not implementing it, on the basis of the RFC's own numbers.

Not closing this issue: keeping it open as the umbrella to track (a)/(b) above. Calling out @NubsCarson — does narrowing this RFC to ASR-only + the small recommender/catalog changes match where you want this to land? If so I think a single follow-up PR closes this out.