runs on cpu when vulkan device is available

[dpblnt]

[ 36%] Building C object CMakeFiles/turboquant.dir/src/backend/vulkan/tq_vulkan_dispatch.c.o
[ 37%] Building C object CMakeFiles/turboquant_shared.dir/src/engine/tq_transformer.c.o
[ 37%] Building C object CMakeFiles/turboquant_shared.dir/src/backend/vulkan/tq_vulkan_dispatch.c.o
[ 38%] Building C object CMakeFiles/turboquant_shared.dir/src/backend/vulkan/tq_vulkan_init.c.o
/opt/TurboQuant.cpp/src/backend/vulkan/tq_vulkan_init.c: In function ‘tq_vk_select_device’:
/opt/TurboQuant.cpp/src/backend/vulkan/tq_vulkan_init.c:201:5: warning: ‘__builtin_strncpy’ output may be truncated copying 255 bytes from a string of length 255 [-Wstringop-truncation]
  201 |     strncpy(g_vk_state.device_name, props.deviceName,
      |     ^
[ 39%] Building C object CMakeFiles/turboquant.dir/src/backend/vulkan/tq_vulkan_init.c.o
/opt/TurboQuant.cpp/src/backend/vulkan/tq_vulkan_init.c: In function ‘tq_vk_select_device’:
/opt/TurboQuant.cpp/src/backend/vulkan/tq_vulkan_init.c:201:5: warning: ‘__builtin_strncpy’ output may be truncated copying 255 bytes from a string of length 255 [-Wstringop-truncation]
  201 |     strncpy(g_vk_state.device_name, props.deviceName,
      |     ^
[ 39%] Linking C static library libturboquant.a
[ 40%] Linking C shared library libturboquant.so
[ 40%] Built target turboquant_shared
[ 40%] Built target turboquant

./quant /mnt/models/gemma-3-4b-it-Q8_0.gguf -p "hello"
Loading model from /mnt/models/gemma-3-4b-it-Q8_0.gguf...
tq_load_model: detected GGUF format
tq_load_gguf: GGUF v3, 444 tensors, 40 metadata keys
tq_load_gguf: architecture = 'gemma3'
tq_load_gguf: hybrid attention detected — sliding head_dim=128 (metadata: 256)
tq_load_gguf: Gemma family detected (sliding_window=1024)
tq_load_gguf: config — layers=34, dim=2560, heads=8/8, head_dim=128, vocab=0
tq_load_gguf: Gemma norms already adjusted (mean=8.3, skipping +1.0)
tq_load_gguf: loaded 34 layers (34 self_attn), dim=2560, heads=8/8, vocab=262208
tq_load_gguf: load-time Q4 conversion enabled (est FP32 = 11.3 GB < 8 GB threshold)
tq_quantize_weights_q4: quantized to Q4 (1806 MB, was ~14450 MB FP32)
tq_load_gguf: Q4 conversion complete — fast matmul path active
tq_load_gguf: madvise(MADV_DONTNEED) on 3.8 GB mmap
Model: 34 layers, dim=2560, heads=8/8, head_dim=128, vocab=262208, inter=10240
KV cache type: uniform_4b, V quant: FP16
tq_load_tokenizer_from_tqm: not a TQM file
tq_load_tokenizer_from_gguf: loaded 262208 tokens (max_len=48)
Loaded tokenizer from GGUF metadata
Threads: 4
Prompt: hello
---
लिटdomமாறு takeaway ი굿ிக்கIssˇoundorestation nutshell Mansfield landing ungroup Terr =~عنی заня Medal^C

topped up cpu, left gpu idle

[unamedkr]

Thanks for the report. Two separate issues here:

1. Vulkan GPU not used for inference

This is a known limitation. The Vulkan backend currently only implements KV cache quantization kernels (quantize/dequantize/attention), not the full forward pass (matmul, RMSNorm, SwiGLU, etc.). The main inference loop runs on CPU regardless of Vulkan availability. GPU offload for the full forward pass is not yet implemented.

Current Vulkan status: compute shaders compile and dispatch works, but the engine doesn't route the core matmul/FFN workload to GPU. This is on the roadmap but not near-term — our priority is the Metal backend for Apple Silicon.

2. Garbage output on Gemma 3 4B

The garbled text is likely the same layer_output_scale bug we recently fixed for Gemma 4. Gemma 3 also uses per-layer output scaling, and our implementation was applying it to the entire hidden state instead of just the layer's residual contribution. This was fixed in commit aaa87c2 — can you try pulling latest main?

If garbage persists after updating, please share the first few lines of output with TQ_DEBUG=1 ./quant model.gguf -p "hello" -n 5 — that will help us trace where the values go wrong.

3. strncpy warning

Fixed in latest main (replaced with memset + memcpy).

[unamedkr]

Update: pushed two changes that should help.

Vulkan GPU auto-activation: When built with -DTQ_BUILD_VULKAN=ON and a Vulkan device is present, quant.cpp now automatically initializes Vulkan and routes KV cache quantization/attention to GPU compute shaders. You should see:

quant.cpp Vulkan: Initialized on <your GPU> (subgroup size 32)
  Vulkan: GPU-accelerated quantize for polar_3b
  Vulkan: GPU-accelerated attention for polar_3b
  ...

The main matmul (weight × activation) still runs on CPU — GPU offload for the full forward pass is a bigger project. But KV cache operations (quantize on store, dequantize + attention on query) now dispatch to Vulkan when available.

strncpy warning: fixed.

Gemma 3 garbage output: Please try pulling latest main. We fixed the layer_output_scale application (was incorrectly scaling the entire hidden state instead of just the layer's contribution). This affects both Gemma 3 and 4.

git pull && cmake --build build -j$(nproc)
./quant model.gguf -p "hello" -n 20

If output is still garbled, run with TQ_DEBUG=1 and share the first few debug lines.