Skip to content

feat(adapter/local-llm): Ollama / llama.cpp / vLLM skill runtime for sovereign and air-gapped pilots #315

Description

@potiuk

Goal: Define and land a skill runtime that works against local LLMs (Ollama, llama.cpp, vLLM, LM Studio, Jan). Grounds: RFC-AI-0004 Principle 3 — Vendor neutrality, which explicitly names "a local-Ollama wrapper" as a target runtime, and RFC-AI-0004's "no cloud-only skills" consequence.

Why local LLM:

  • Sovereign deployments — projects bound by data-residency rules (EU, government, defence-orbit OSS) that can't send issue / mail content to a US cloud API
  • Air-gapped triage — security teams that need to assess <security-list> traffic without an internet connection
  • Cost ceiling — projects unable to commit to a per-token cloud bill, but happy to run a 70B model on shared infra

What "parity" means (calibrated for what local LLMs can realistically do):

  • Skills under .claude/skills/<name>/SKILL.md are invokable via a local-LLM-backed agent loop
  • The tools/* bridges are reachable — these are language-agnostic CLI calls, no LLM context needed
  • Sandbox / HITL primitives map to the same bubblewrap baseline; the local-LLM runtime is the new piece
  • Realistic scoping: some skills (multi-step reasoning over long contexts — security-issue-triage, pr-management-code-review) may need a high-capability local model (Llama 70B, Qwen 72B, DeepSeek). Document the per-skill model-size floor so adopters can pick a model that actually works.

Suggested approach:

  • Pick an agent-loop frontend that supports local LLMs natively — candidates: aider --model ollama/..., goose with a local backend, continue.dev, a fresh thin wrapper around llama.cpp's OpenAI-compatible API
  • Document the model-size floor empirically (per skill, against the existing eval suite at tools/skill-evals/)
  • Land a setup-local-llm skill family alongside setup-isolated-* for the runtime-side install

Reference:

Metadata

Metadata

Assignees

No one assigned

    Labels

    capability:platformFramework / agent substrate skills (install, verify, doctor, override, status, setup bootstrap)enhancementNew feature or requestfamily:toolstools/*good first issueGood for newcomers

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions