[Bug]: llama.cpp server missing --gpu-layers flag, model runs on CPU despite GPU available

[BreakNRetest]

CoPaw Version v1.0.0

Bug Description

When using CoPaw's built-in local model (agentscope-ai/CoPaw-Flash-4B-Q8_0), the llama.cpp server is launched without GPU acceleration. The model runs entirely on CPU despite a compatible NVIDIA GPU being available.

Steps to Reproduce

1. Enable a local llama.cpp model in CoPaw (e.g. agentscope-ai/CoPaw-Flash-4B-Q8_0)
2. Start CoPaw
3. Observe that the llama.cpp server process runs on CPU only
4. Check nvidia-smi — GPU utilization remains at ~0% during inference

Expected Behavior

The llama.cpp server should automatically detect and use the available NVIDIA GPU by passing --gpu-layers N (or -ngl N), offloading model layers to VRAM for GPU-accelerated inference.

Actual Behavior

The llama.cpp server launches without any GPU offloading flags:

llama-server.exe --host 127.0.0.1 --port 59570 --model ... --alias agentscope-ai/CoPaw-Flash-4B-Q8_0

No --gpu-layers parameter is passed, even when:

• A compatible NVIDIA GPU is present (RTX 5070 Ti, 16GB VRAM)
• CUDA 13.2 is installed and working
• The llama.cpp binary has CUDA support compiled in (verified via --help)

Environment

• OS: Windows 11
• GPU: NVIDIA GeForce RTX 5070 Ti (16GB VRAM)
• CUDA Version: 13.2
• Driver: 595.97
• CoPaw Version: (latest, installed via pip/Python 3.11)
• llama.cpp binary: C:\Users\edtor\.copaw\local_models\bin\llama-server.exe

Root Cause

In copaw/local_models/llamacpp.py, the _create_server_process() method builds the llama-server command with only:

command = [
    str(self.executable),
    "--host", "127.0.0.1",
    "--port", str(port),
    "--model", str(resolved_model_path),
    "--alias", model_name,
]

No --gpu-layers or -ngl flag is added, and there is no configuration option to enable GPU offloading.

Suggested Fix

Add GPU layer offloading to the llama-server command. Options:

1. Auto-detect GPU and set optimal layers — detect NVIDIA GPU and add --gpu-layers -1 (all layers) or a calculated value
2. Add a config option — allow users to set gpu_layers in the local model config

Example fix:

command = [
    str(self.executable),
    "--host", "127.0.0.1",
    "--port", str(port),
    "--model", str(resolved_model_path),
    "--alias", model_name,
]
# Add GPU layers if NVIDIA GPU is available
if self._has_nvidia_gpu():
    command.extend(["--gpu-layers", "35"])  # or -1 for all layers

Logs

INFO ... | Setting up llama.cpp server for model agentscope-ai/CoPaw-Flash-4B-Q8_0 at path C:\Users\edtor\.copaw\local_models\models\agentscope-ai\CoPaw-Flash-4B-Q8_0\CoPaw-flash-4B-20260330-q8.gguf
INFO ... | llama.cpp health check returned 503 while waiting for http://127.0.0.1:59570/health
...
INFO ... | llama.cpp server is ready
INFO ... | llama.cpp server started on port 59570 for model agentscope-ai/CoPaw-Flash-4B-Q8_0

Note: Health check returns 503 during startup — likely due to slow CPU-only model loading.

Additional Context

• The llama.cpp binary at C:\Users\edtor\.copaw\local_models\bin\llama-cli.exe --help confirms GPU support flags exist: -ngl, --gpu-layers, --n-gpu-layers N
• Other local model runners (Ollama, LM Studio) correctly use GPU acceleration on the same system

[github-actions]

Welcome to CoPaw! 🐾

Hi @BreakNRetest, thank you for your first issue!

We'll review your issue soon. Thank you for supporting CoPaw!

[pan-x-c]

Thanks for your feedback. Due to differences in the number of layers and the size of each layer across different LLM, GPU support will be rolled out gradually in subsequent versions.