Skip to content

fix(llama-cpp): preserve GPU layers during option passthrough - #11193

Merged
mudler merged 1 commit into
masterfrom
bot/issue-11191-generic-passthrough
Jul 30, 2026
Merged

fix(llama-cpp): preserve GPU layers during option passthrough#11193
mudler merged 1 commit into
masterfrom
bot/issue-11191-generic-passthrough

Conversation

@localai-org-maint-bot

Copy link
Copy Markdown
Collaborator

Description

Fixes #11191.

The latest llama.cpp argument parser asserts that main and draft n_gpu_layers still contain negative initialization sentinels when parser options are registered. LocalAI resolves those fields from the model request before applying generic passthrough flags, so any passthrough option could abort the backend and surface as an EOF from the frontend.

This stages the parser sentinels while applying raw flags, then restores LocalAI's resolved GPU-layer values unless the passthrough explicitly supplies a main or draft GPU-layer flag. Explicit auto remains authoritative.

Notes for Reviewers

  • Added standalone regression coverage for staging/restoration, numeric overrides, and explicit auto.
  • Verified with NLOHMANN_INCLUDE=backend/cpp/llama-cpp/llama.cpp/vendor bash backend/cpp/run-unit-tests.sh (3 test files).
  • Verified the backend preparation path copies the helper and test into the generated grpc-server tree.
  • Full backend compilation was not run because it is a long build; CI covers the backend matrix.

Signed commits

  • Yes, I signed my commits.
  • Documentation updated (docs/content/) for user-facing changes, or not applicable

Stage the negative GPU-layer sentinels expected by the upstream argument parser, then restore LocalAI resolved values unless a passthrough flag explicitly overrides them. This avoids the parser assertion that terminated the backend for any generic option.

Assisted-by: Codex:gpt-5
@mudler

mudler commented Jul 30, 2026

Copy link
Copy Markdown
Owner

@localai-org-maint-bot did you reproduced it locally?

@localai-org-maint-bot

Copy link
Copy Markdown
Collaborator Author

Yes, but at the focused parser-path level rather than through a full LocalAI backend build. I reproduced the exact llama.cpp assertion locally by exercising the option-registration path with pre-resolved n_gpu_layers, then verified the staging/restoration fix with the standalone C++ regression suite. I did not compile and launch the complete llama-cpp backend end to end; the PR description calls that out explicitly.

@mudler
mudler merged commit 5e54189 into master Jul 30, 2026
95 of 96 checks passed
@mudler
mudler deleted the bot/issue-11191-generic-passthrough branch July 30, 2026 13:54
@localai-bot localai-bot added the bug Something isn't working label Aug 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Generic option passthrough doesn't work

3 participants