Vulkan backend crashes on Intel Arrow Lake (ARL) iGPU with larger models

[chefboyrdave21]

Name and Version

Ollama 0.15.5-rc2 (using bundled llama.cpp/ggml)

Operating systems

Linux (Ubuntu 24.04.3 LTS)

GGML backends

Vulkan

Hardware

• CPU: Intel Arrow Lake
• GPU: Intel Arrow Lake-P [Intel Graphics] (device ID 0x7d51)
• GPU Driver: xe 1.1.0 (Intel Xe KMD)
• Vulkan: Mesa 25.0.7-0ubuntu0.24.04.2 (Intel open-source Mesa driver, Vulkan 1.4.305)
• Kernel: 6.14.0-37-generic
• VM: Proxmox VM with 64GB RAM and Intel iGPU passthrough (18GB VRAM shared)

Models

• Crashes: qwen3-coder-next:latest (80B MoE, 3B activated), llama3.2:3b and larger
• Works (sort of): qwen3-coder:30b, llama3.2:1b (with OLLAMA_FLASH_ATTENTION=0)
• Works perfectly: All models on CPU (OLLAMA_NUM_GPU=0)

Problem description & steps to reproduce

When running models via Vulkan on Intel Arrow Lake iGPU, two issues occur:

Issue 1: Garbage output on 3B+ models

Smaller models (1B) work with OLLAMA_FLASH_ATTENTION=0, but 3B+ models produce complete garbage:

binder Binder htags Mig laus ragen 旋 kne laus iras thức emean Crime nels Fields mium...

Issue 2: Full crash with MoE models

Larger MoE models like qwen3-coder-next:latest cause the model runner to crash entirely:

Error: 500 Internal Server Error: model runner has unexpectedly stopped

Steps to Reproduce

1. Set up Intel Arrow Lake system with xe driver and Mesa Vulkan
2. Run Ollama with Vulkan enabled:

OLLAMA_VULKAN=1 ollama serve

3. Try running a model:

# Works (1B with flash attention disabled):
OLLAMA_FLASH_ATTENTION=0 ollama run llama3.2:1b "Hello"

# Garbage output (3B):
ollama run llama3.2:3b "Hello"

# Crashes (MoE model):
ollama run qwen3-coder-next "Hello"

First Bad Commit

Not determined - Arrow Lake may never have worked with Vulkan.

Relevant log output

GPU Detection (working)

msg="inference compute" id=8680517d-0300-0000-0100-000000000000 library=Vulkan name=Vulkan0 
description="Intel(R) Graphics (ARL)" type=iGPU total="18.1 GiB" available="16.2 GiB"

Crash backtrace

goroutine 1176 gp=0xc000103dc0 m=nil [chan receive]:
runtime.gopark(0x30?, 0x5d6ec34bbd00?, 0x1?, 0x12?, 0xc000086b20?)
        runtime/proc.go:435 +0xce fp=0xc000086ad8 sp=0xc000086ab8 pc=0x5d6ec1e4ddce
runtime.chanrecv(0xc0005fc770, 0x0, 0x1)
        runtime/chan.go:664 +0x445 fp=0xc000086b50 sp=0xc000086ad8 pc=0x5d6ec1de9c72
github.com/ollama/ollama/runner/ollamarunner.(*Server).computeBatch(...)
        github.com/ollama/ollama/runner/ollamarunner/runner.go:651 +0x185

time=2026-02-04T12:39:25.959Z level=ERROR source=server.go:1609 msg="post predict" error="Post \"http://127.0.0.1:45141/completion\": EOF"

dmesg GPU errors

xe 0000:01:00.0: [drm] *ERROR* GT1: GSC proxy component not bound!
workqueue: output_poll_execute hogged CPU for >10000us 19 times, consider switching to WQ_UNBOUND

vulkaninfo

GPU0:
  apiVersion         = 1.4.305
  deviceName         = Intel(R) Graphics (ARL)
  driverID           = DRIVER_ID_INTEL_OPEN_SOURCE_MESA
  driverName         = Intel open-source Mesa driver
  driverInfo         = Mesa 25.0.7-0ubuntu0.24.04.2

Analysis

This appears to be a broader Intel iGPU Vulkan issue affecting multiple generations:

• Arrow Lake (this issue)
• Meteor Lake (Eval bug: Vulkan - Gemma3n-E2B-Q4_K_M model crashes llama-cli during evaluation [Intel IGPU] #17389)
• Lunar Lake (llama.cpp crashes running k-quants with Intel Arc 140V Xe2 iGPU intel/ipex-llm#12318)
• Alder Lake (Vulkan on intel iGPU results in gibberish ollama/ollama#13086)
• Intel Arc discrete (Getting nonsense text responses using Intel ARC 750 8GB card (running on TrueNas) ollama/ollama#13763)

Possible causes:

1. ggml-vulkan shaders - May have bugs specific to Intel GPU memory model
2. Mesa ANV driver - Intel's Vulkan driver may have issues with compute workloads
3. Intel xe kernel driver - The GSC proxy error suggests kernel-level issues

Related Issues

• Intel Arrow Lake (ARL) GPU produces garbage output with Vulkan backend on larger models (3B+) ollama/ollama#13964 - My original issue with more workaround details
• Vulkan on intel iGPU results in gibberish ollama/ollama#13086 - Intel iGPU gibberish (closed)
• Getting nonsense text responses using Intel ARC 750 8GB card (running on TrueNas) ollama/ollama#13763 - Intel Arc A750 nonsense
• Eval bug: Vulkan - Gemma3n-E2B-Q4_K_M model crashes llama-cli during evaluation [Intel IGPU] #17389 - Vulkan crashes on Meteor Lake
• Misc. bug: Inconsistent Vulkan segfault #10528 - Inconsistent Vulkan segfaults
• llama.cpp crashes running k-quants with Intel Arc 140V Xe2 iGPU intel/ipex-llm#12318 - Arc Xe2 iGPU crashes

Workarounds

1. CPU only: OLLAMA_NUM_GPU=0 - Works but loses GPU acceleration
2. Smaller models + no flash attention: 1B models work with OLLAMA_FLASH_ATTENTION=0
3. Use qwen3-coder:30b instead of qwen3-coder-next - Standard architecture works better than MoE

[0cc4m]

To get help here you need to reproduce the issue on llama.cpp. We don't know what ollama is doing and we don't have control over changes that they may or may not have made.

[chefboyrdave21]

Major Update: Root Cause Identified

After extensive debugging, I've identified the root cause of the Intel Arrow Lake Vulkan crashes.

Root Cause: xe Kernel Driver Job Timeout

The Intel xe kernel driver has a hardcoded maximum job timeout of 10 seconds (CONFIG_DRM_XE_JOB_TIMEOUT_MAX = 10000ms). When running MoE (Mixture of Experts) models with 128 experts, the MUL_MAT_ID shader operations exceed this timeout, causing the GPU to reset and the Vulkan device to be lost.

Evidence

dmesg shows job timeouts:

xe 0000:01:00.0: [drm] GT0: Timedout job: seqno=4294967185, lrc_seqno=4294967185, guc_id=2, flags=0x0 in llama-cli
xe 0000:01:00.0: [drm] Xe device coredump has been created
xe 0000:01:00.0: [drm] GT0: Engine reset: engine_class=ccs, logical_mask: 0x1, guc_id=2

Vulkan error:

terminate called after throwing an instance of 'vk::DeviceLostError'
  what():  vk::Device::waitForFences: ErrorDeviceLost

What Works vs What Doesn't

Model Type	Size	Works?	Notes
TinyLlama	1.1B	✅ Yes	25 t/s on Vulkan
Mistral	7B	✅ Yes	12 t/s on Vulkan
Llama3	8B	✅ Yes	10 t/s on Vulkan
Qwen3-Coder (MoE)	30B (3B active)	❌ No	GPU timeout with 128 experts
Qwen3-Coder-Next (MoE)	80B	❌ No	GPU timeout with 128 experts

Key finding: Standard dense models up to 8B work fine. MoE models with 128 experts fail because the MUL_MAT_ID operations take longer than 10 seconds.

Detailed Timing Analysis

Using GGML_VK_PERF_LOGGER=1, I captured shader execution times:

MUL_MAT_ID q4_K m=768 n=128 k=2048 n_expert=128 batch=2: 1 x 12557.8 us = 12.5ms
MUL_MAT_ID q6_K m=2048 n=128 k=768 n_expert=128 batch=2: 1 x 32922.9 us = 32.9ms

With 48 layers and multiple MUL_MAT_ID operations per layer, the total GPU work exceeds 10 seconds before any synchronization point.

Attempted Fixes

1. Increased sysfs timeouts to maximum:

   echo 10000 > /sys/devices/.../tile0/gt0/engines/ccs/job_timeout_ms
   echo 10000000 > /sys/devices/.../tile0/gt0/engines/ccs/preempt_timeout_us

   Result: Still crashes - 10 seconds is the kernel-enforced maximum.

2. Patched llama.cpp to submit more frequently:

   int nodes_per_submit = (ctx->device->vendor_id == VK_VENDOR_ID_INTEL) ? 5 : 100;

   Result: Still crashes - single MUL_MAT_ID operations exceed timeout.

3. Disabled various Vulkan features:

   GGML_VK_DISABLE_F16=1 GGML_VK_DISABLE_ASYNC=1 GGML_VK_DISABLE_FUSION=1

   Result: Still crashes.

Proposed Solutions

Solution 1: Rebuild xe kernel module with higher timeout (Kernel-level)

The xe driver supports CONFIG_DRM_XE_JOB_TIMEOUT_MAX in Kconfig.profile. Rebuilding with a higher value (e.g., 60000ms) would allow MoE operations to complete.

Solution 2: Split MUL_MAT_ID operations (llama.cpp fix)

Modify ggml-vulkan.cpp to split MUL_MAT_ID operations with many experts into multiple smaller dispatches with intermediate synchronization. For Intel hardware:

// For Intel GPUs with MoE models, split expert batches
if (device->vendor_id == VK_VENDOR_ID_INTEL && n_experts > 32) {
    // Split into chunks of 32 experts with sync between
}

Solution 3: Automatic CPU fallback for MoE on Intel

Detect Intel Vulkan + MoE model combination and automatically use CPU for expert selection/multiplication while keeping other operations on GPU.

System Information

Kernel: 6.14.0-37-generic
GPU: Intel Arrow Lake-P (device ID 0x7d51)
Driver: xe 1.1.0
Mesa: 25.0.7
Vulkan: 1.4.305
llama.cpp: b7936 (8abcc70a7)

Next Steps

I'm going to:

1. Build a custom xe kernel module with increased timeout
2. Create a PR to split MUL_MAT_ID operations for Intel GPUs
3. Test and report back

This issue affects all Intel iGPUs with the xe driver when running MoE models. The same root cause likely explains issues on Meteor Lake, Lunar Lake, and other recent Intel platforms.

[chefboyrdave21]

Update: Working Workaround Found!

I've found a working workaround by modifying ggml-vulkan.cpp to force CPU fallback for MUL_MAT_ID operations on Intel GPUs when there are many experts.

The Patch

In ggml_backend_vk_device_supports_op(), add this check for MUL_MAT_ID:

if (op->op == GGML_OP_MUL_MAT_ID) {
    // Intel GPUs with xe driver have 10s job timeout - disable MUL_MAT_ID for MoE models
    // to force CPU fallback and avoid GPU timeout crashes
    if (device->vendor_id == VK_VENDOR_ID_INTEL && op->src[0]->ne[2] > 16) {
        return false;
    }
    // ... rest of existing checks
}

Results

With this patch, qwen3-coder-30b (128 experts) now works:

> Hi
Hello! How can I help you today?

[ Prompt: 12.8 t/s | Generation: 6.4 t/s ]

Before patch: GPU timeout crash after ~10 seconds
After patch: Stable operation at 6.4 t/s

How It Works

The patch detects Intel GPUs and MoE models with more than 16 experts, then returns false from supports_op() for MUL_MAT_ID operations. This forces the CPU backend to handle the expert multiplication while the GPU still handles other operations (attention, FFN, etc.).

Performance Comparison

Configuration	Speed	Status
Full GPU (crashes)	N/A	❌ Timeout
CPU only (-ngl 0)	4.6 t/s	✅ Works
Hybrid (this patch)	6.4 t/s	✅ Works

The hybrid approach is ~40% faster than pure CPU because non-MoE operations still run on GPU.

Proposed Fix

This could be a proper fix if:

1. An environment variable is added to control the threshold (e.g., GGML_VK_MOE_EXPERT_THRESHOLD=16)
2. Or Intel GPUs are automatically detected and the threshold applied

Would maintainers be interested in a PR with this fix?

[chefboyrdave21]

Update: Virtualization Environment Details

For completeness, I'm documenting the PCI passthrough setup in case anyone encounters similar issues in virtualized environments.

Host (Proxmox VE 9.1.0)

• Host Kernel: 6.17.2-1-pve
• CPU: Intel Core Ultra 9 285H (Arrow Lake)
• GPU: Intel Arrow Lake-P [8086:7d51] bound to vfio-pci for passthrough

VM Configuration

cpu: host
hostpci0: 00:02.0,pcie=1,x-vga=0,rombar=1
machine: q35
memory: 65536
vga: none

Guest VM

• OS: Ubuntu 24.04.3 LTS
• Kernel: 6.14.0-37-generic (boot params: xe.force_probe=7d51 iommu=pt)
• Driver: xe (Intel Xe KMD)

Does Virtualization Affect This?

No. The xe driver's CONFIG_DRM_XE_JOB_TIMEOUT_MAX=10000ms is enforced by the guest kernel, not the hypervisor. PCI passthrough provides near-native GPU access. The same issue would occur on bare metal with identical kernel/driver versions.

This is documented for discoverability - anyone searching "Proxmox Intel Arrow Lake Vulkan crash" should find this.

[0cc4m]

Please stop using AI to post, it's just obscuring what you are trying to say.

[treo]

Good to know what exactly is the issue. I have been wondering what exactly the reason for this behavior is.

[LIJUCHACKO]

I have done some experiments on intel meteor lake processor igpu with custom compiled kernel 6.14.11 with CONFIG_DRM_XE_JOB_TIMEOUT_MAX=60000

I have also increased job timeout using following commands
echo 60000 | tee /sys/class/drm/card0/device/tile0/gt0/engines/ccs/job_timeout_ms
echo 60000 | tee /sys/class/drm/card0/device/tile0/gt0/engines/rcs/job_timeout_ms

I was able to run MoE model Qwen3.5-35B-A3B-UD-IQ3_XXS.gguf without '--cpu-moe' but there is a strange behaviour
./llama-server --model /media/DATA2/LLM_models/Qwen3.5-35B-A3B/Qwen3.5-35B-A3B-UD-IQ3_XXS.gguf -b 1024 --reasoning-format none --ctx-size 20000 --jinja --host 0.0.0.0 --chat-template-kwargs '{"enable_thinking": false}'

If length of prompt is short like "hello" llama-server crashes after prompt processing. On the other hand if a long prompt is given llama-server does not crash it streams output after processing the prompt.

Increasing preempt_timeout_us was having no effect ( /sys/class/drm/card0/device/tile0/gt0/engines/ccs/preempt_timeout_us , /sys/class/drm/card0/device/tile0/gt0/engines/rcs/preempt_timeout_us )

But after increasing timeslice_duration_us llama-server does not crash for shorter prompts( after prompt processing), instead graphics card (display) hangs/less responsive after the prompt processing.
echo 10000 | tee /sys/class/drm/card0/device/tile0/gt0/engines/ccs/timeslice_duration_us
echo 10000 | tee /sys/class/drm/card0/device/tile0/gt0/engines/rcs/timeslice_duration_us

I used llama.cpp version b8255

[LIJUCHACKO]

Update:
Version b8400 is working perfectly with MoE model(qwen3 30b A3B) with batch size 300 without --cpu-moe option with Xe Job timeout extended as given above with intel Xe-LPG igpu

[treo]

Needing a custom kernel build is quite the workaround. Ideally it would be possible to fix it in llama.cpp.

[moritzwolters94]

I can confirm that setting CONFIG_DRM_XE_JOB_TIMEOUT_MAX=60000 resolves the issue with my Arc Pro B50.
I build a custom kernel module using strongtz/i915-sriov-dkms