fix: segmentation fault on Vulkan (gfx1031 radv)

[Skyrion9]

• Calling prefill_fast causes segmentation faults in Archlinux when using on an RX 6700 (Latest radv as of this PR).

• Reproducible by loading any layers to GPU when using Vulkan. YMMV between different GPU architectures and driver versions, this was the only way to successfully generate for me.

• 0 layers on GPU or compiling for CPU backend fixes/works around this, but that essentially disables GPU benefit.

Summary by CodeRabbit

• Bug Fixes
  • Updated text generation to use the standard prefill process, improving consistency and reliability during prompt processing.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 90472928-0662-46a3-a3d3-dd8ab82e375d

📥 Commits

Reviewing files that changed from the base of the PR and between 2c33261 and 56929b9.

📒 Files selected for processing (1)

• src/s2_generate.cpp

---

📝 Walkthrough

Walkthrough

Generation now calls model.prefill(...) instead of model.prefill_fast(...) during prefill. Existing failure handling, early return behavior, timing, and decoding flow remain unchanged.

Changes

Generation prefill

Layer / File(s)	Summary
Switch generation prefill call src/s2_generate.cpp	The generation prefill path invokes model.prefill(...) while retaining the existing failure check and early return.

Estimated code review effort: 1 (Trivial) | ~2 minutes

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title matches the main change: a Vulkan-related segfault fix in generation/prefill behavior.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[Skyrion9]

Upon further investigation this appears to be an OOM issue.

• In Linux RADV or MESA drivers the symptom is segfault or "device lost" instead of your usual vulkan OOM errors.

• This is proven by using --codec-cpu in which case segfault doesn't occur even without this fix.

• Fast prefilling for the codec itself can easily cause OOM especially in GPU backends

• Chunking can be a good idea in general since it allows us to run with lower memory budgets, and in the case of using --codec-cpu it hits the same codepath as fast_prefill when it matters anyways return eval_cached(flat_tokens, n_tokens, n_threads, result);

I think chunking on the CPU would do more harm than good, so despite this realization I think the current PR is optimal. However, for very long generations (Like entire pages, audiobook-type load) this CPU chunking might be desirable also if the allocations are heavy.