Skip to content

LLM Setup

Jump to bottom
blackbuble edited this page Oct 30, 2026 · 1 revision

LLM Configuration

PentestAI supports 15 LLM providers with automatic routing based on task complexity. One env var is all it takes to get started.

Quick Start

# Minimal — single provider
export ANTHROPIC_API_KEY=sk-ant-...
python3 -m pentestai

# Optimal — multi-tier routing
export ANTHROPIC_API_KEY=sk-ant-...   # complex reasoning
export GROQ_API_KEY=gsk_...           # fast/cheap parsing
export DEEPSEEK_API_KEY=...           # balanced reasoning, very cheap

# Local — no API key
ollama pull llama3.1
# PentestAI auto-detects Ollama on localhost:11434

build_router_from_env() automatically detects all available providers from env vars and builds a fleet with routing based on task tier.

Inside Docker Compose

docker-compose.yml forwards the entire contents of .env via env_file: .env, so all providers in the table below only need to be filled in .env, then:

docker compose up -d --force-recreate pentestai   # env is read when the container is created
docker compose exec -T pentestai python3 -c \
  "from core.llm import build_router_from_env; print(build_router_from_env().status())"

Providers with an empty key are skipped automatically. scripts/env_test.py labels an empty key as FAIL — that is a configuration status, not a container error.

Status as of 2026-10-24: Anthropic and Ollama (qwen2.5:3b) are live-verified. The 11 cloud providers with API-key env vars have no keys in .env yet, so they are skipped.

All Supported Providers

Provider Env Var Default Model Tier Notes
Anthropic ANTHROPIC_API_KEY claude-sonnet-4-6 complex Recommended for deep reasoning
OpenAI OPENAI_API_KEY gpt-4o complex
xAI Grok XAI_API_KEY grok-3 complex
Mistral MISTRAL_API_KEY mistral-large-latest medium
DeepSeek DEEPSEEK_API_KEY deepseek-chat medium Very cheap, strong reasoning
Groq GROQ_API_KEY llama-3.3-70b-versatile simple Very low latency
Google Gemini GOOGLE_API_KEY gemini-2.0-flash simple
Perplexity PERPLEXITY_API_KEY llama-3.1-sonar-large-128k-online simple Online models, web search
Cohere COHERE_API_KEY command-r-plus medium
Together AI TOGETHER_API_KEY meta-llama/Llama-3-70b-chat-hf simple 50+ open-source models
OpenRouter OPENROUTER_API_KEY anthropic/claude-sonnet-4-5 all 100+ models via a single key
Azure OpenAI AZURE_OPENAI_API_KEY + AZURE_OPENAI_ENDPOINT gpt-4o complex Enterprise, also requires an endpoint URL
Ollama — (auto-detect) llama3.2 simple Fully local, no API key
9router NINEROUTER_API_KEY auto all Local gateway, 40+ providers
Custom CUSTOM_LLM_API_KEY default — Any OpenAI-compatible endpoint

Azure OpenAI — Special Configuration

Azure requires three variables:

export AZURE_OPENAI_API_KEY=...
export AZURE_OPENAI_ENDPOINT=https://your-resource.openai.azure.com
export AZURE_OPENAI_DEPLOYMENT=gpt-4o   # deployment name (optional)

Tier Routing

PentestAI automatically selects a model based on task complexity:

Tier Task Ideal Provider
simple Parsing output, classification, info extraction Groq, Ollama, Gemini
medium Vulnerability assessment, enumeration DeepSeek, Mistral, Cohere
complex Attack planning, exploit reasoning, report Anthropic, OpenAI, xAI
all All tasks (override) OpenRouter, 9router 
# CLI
llm                           # view status and active models
llm add                       # wizard to add a model
llm tier claude-sonnet-4-6 complex  # change tier
llm routing on|off            # enable/disable routing
llm test                      # test connections for all models

SDK Compatibility

Anthropic SDK 1.15.0 — the installed version. Streaming uses __stream_text__() with a fallback to get_final_text():

with client.messages.stream(...) as s:
    for text in s.__stream_text__():   # SDK 1.15.0
        print(text, end="")

Non-streaming: resp.content[0].text or iterate over resp.content to find the block of type text.

All other providers use an OpenAI-compatible endpoint via openai.OpenAI(base_url=...).

Offline / Air-gapped

# Install Ollama
curl -fsSL https://ollama.ai/install.sh | sh
ollama pull llama3.1:70b    # better reasoning

# PentestAI auto-detects, no env var needed
python3 -m pentestai

The LLM is optional — all tools (nmap, gobuster, sqlmap, etc.) run without an LLM. The agent layer (ReAct, KillChain) requires an LLM.

Manual Configuration (llm_fleet.json)

The sessions/llm_fleet.json file stores persistent configuration:

{
  "models": [
    {
      "provider_id": "anthropic",
      "model_name": "claude-sonnet-4-6",
      "api_key": "sk-ant-...",
      "tier": "complex",
      "enabled": true,
      "display_name": "Claude Sonnet"
    },
    {
      "provider_id": "groq",
      "model_name": "llama-3.3-70b-versatile",
      "api_key": "gsk_...",
      "tier": "simple",
      "enabled": true,
      "display_name": "Llama 3.3 Fast"
    }
  ],
  "routing_enabled": true,
  "fallback_to_any": true
}

Valid provider IDs: anthropic, openai, groq, ollama, openrouter, google, mistral, xai, deepseek, together, perplexity, azure, cohere, 9router, custom

Circuit Breaker

If a model fails 3× in a row, the circuit breaker opens (OPEN state) and falls back to another provider. Recovery after 60 seconds. Configured in core/retry.py.

Validation Notes

All 15 providers have been unit tested (with mock HTTP). Live calls are verified for Anthropic and Ollama only; run python3 scripts/live_llm_check.py after filling in more keys. The script skips providers that are not configured and never prints secrets.

Custom Endpoint & WorkBuddy

For any provider not in the list — including WorkBuddy, LM Studio, vLLM, llama.cpp server, or an internal API — use the custom provider.

WorkBuddy

WorkBuddy exposes an endpoint whose format needs to be checked first. Two possibilities:

If WorkBuddy has an OpenAI-compatible endpoint:

export CUSTOM_LLM_BASE_URL=https://api.workbuddy.ai/v1   # or the URL provided
export CUSTOM_LLM_API_KEY=<workbuddy-api-key>
export CUSTOM_LLM_MODEL=<model-name>
export CUSTOM_LLM_NAME=WorkBuddy
export CUSTOM_LLM_TIER=complex
python3 __main__.py

Or via the interactive shell:

PentestAI> llm add
Provider ID: custom
Base URL:    https://api.workbuddy.ai/v1
API Key:     <workbuddy-key>
Model:       <model-name>
Tier:        complex
Name:        WorkBuddy

Verify the connection:

PentestAI> llm test

How to tell whether an endpoint is OpenAI-compatible

Test with curl:

curl -X POST https://api.workbuddy.ai/v1/chat/completions \
  -H "Authorization: Bearer <key>" \
  -H "Content-Type: application/json" \
  -d '{"model":"<model>","messages":[{"role":"user","content":"hello"}],"max_tokens":10}'

If the response is formatted as {"choices":[{"message":{"content":"..."}}]} → compatible.

Other providers via custom

Tool Base URL Notes
LM Studio http://localhost:1234/v1 Local, no API key
vLLM http://localhost:8000/v1 Local server
llama.cpp http://localhost:8080/v1 Local server
Ollama Auto-detect No manual config needed
LiteLLM proxy http://localhost:4000 Multi-provider proxy
Any OpenRouter alt https://openrouter.ai/api/v1 Uses OPENROUTER_API_KEY

Env vars for a custom endpoint

Env Var Default Description
CUSTOM_LLM_BASE_URL — Endpoint URL (required for auto-detect)
CUSTOM_LLM_API_KEY no-key-needed API key (optional if no auth is required)
CUSTOM_LLM_MODEL default Name of the model to use
CUSTOM_LLM_NAME Custom LLM Display name in the shell
CUSTOM_LLM_TIER complex Routing tier

Home

Getting Started

Core

Scanning

Reporting

Project

Clone this wiki locally