feat: Add GGUF/llama-cpp backend to terraphim LLM proxy layer

[AlexMikhalev]

Problem

terraphim-ai has no local inference capability. All LLM calls go through remote APIs (OpenRouter via genai). For machines without GPU or API access, there is no fallback. GGUF models for MedGemma are available (unsloth/medgemma-1.5-4b-it-GGUF, 11.8K downloads on HuggingFace).

Proposed Change

Add a terraphim_llm_local crate (or feature in terraphim_multi_agent) that wraps llama-cpp-rs for local GGUF inference. Implement the same LLM client trait so agents can transparently use local or remote models.

Key requirements:

• CPU-only inference support (many dev machines lack GPU)
• Automatic GGUF model download via hf-hub crate
• Quantization variant selection (Q4_K_M ~2.5GB for 4B models, Q8_0 for higher quality)
• Same trait interface as the remote genai client for seamless swapping

Scope

• New crate crates/terraphim_llm_local/ or feature gate in terraphim_multi_agent
• Dependencies: llama-cpp-rs, hf-hub (both approved)

Context

This is UPLIFT-5 from the medgemma-competition multi-agent integration plan. Local GGUF inference is essential for development workflows where remote API calls are slow or unavailable. The MedGemma 1.5-4b-it GGUF model is the primary target for local inference.

Related upstream issues: #534, #535, #536

[AlexMikhalev]

Verification and Validation Report: Issue #538

Date: 2026-03-11
Status: NOT IMPLEMENTED
Decision: NO-GO (feature request not yet implemented)

---

Executive Summary

Issue #538 proposes adding local GGUF inference capability via llama-cpp-rs. The current codebase supports remote (OpenRouter) and local (Ollama) providers, but has no direct GGUF/llama.cpp integration.

Requirements Verification

Requirement	Status	Evidence
terraphim_llm_local crate exists	NOT MET	No crate found
llama-cpp-rs dependency	NOT MET	Not in any Cargo.toml
hf-hub dependency	NOT MET	Not in any Cargo.toml
CPU-only inference support	NOT MET	No GGUF implementation
Same trait interface as remote	MET	LlmClient trait exists

Key Findings

Existing Architecture: The codebase has a well-designed LlmClient trait at crates/terraphim_service/src/llm.rs:31-56 that already supports multiple providers:

• OpenRouterClient - remote API
• OllamaClient - local via HTTP
• RouterBridgeLlmClient - intelligent routing

Alternative Available: Ollama integration provides local inference capability. Users can run GGUF models through Ollama:

{
  "extra": {
    "llm_provider": "ollama",
    "ollama_model": "llama3.1",
    "ollama_base_url": "http://127.0.0.1:11434"
  }
}

Options

1. Implement GGUF support (2-3 days effort): Create terraphim_llm_local crate with llama-cpp-rs, hf-hub for model download, and runtime quantization selection

2. Use Ollama alternative: Ollama already supports importing and serving GGUF models - no code changes needed

3. Close as not planned: If Ollama meets the use case

Recommendation

If the requirement is "local inference without remote APIs", the Ollama integration already provides this. If the requirement is specifically "direct GGUF file loading without Ollama dependency", then implementation work is needed.

---

Full report: .docs/verification-validation-report-issue-538.md