LLM Agents

LLM Agents — Complete List of Supported Models

PentestAI uses JSON structured output by default, so almost any model that can emit valid JSON can drive the loop. Native tool calling is also available for Anthropic and OpenAI via complete_with_tools() (PENTESTAI_NATIVE_TOOLS=1, the default); other providers fall back to the JSON plan.

---

How LLMs Work in PentestAI

ReActAgent._reason() 
  → send REACT_SYSTEM prompt + observations to the LLM
  → LLM returns JSON: {action, action_args, reasoning, risk_level, ...}
  → parse JSON → execute action → observe → repeat

The model does not need to support tool calling. What is required: consistent JSON output and adequate security knowledge.

---

Tier Routing

PentestAI automatically selects a model based on task complexity:

Tier	Used for	Model criteria
complex	Attack planning, exploit reasoning, ReAct loop	High JSON consistency, good security knowledge
medium	Vulnerability assessment, finding triage	Moderate JSON, adequate reasoning
simple	Parse output, classify findings, quick lookup	Fast, cheap, basic JSON
all	All tasks (override, gateway)	Routing delegated to the provider

---

Native Providers (auto-detected from env vars)

Tier Complex — Recommended for the ReAct loop

Provider	Default Model	Setup	JSON	Security
Anthropic	claude-sonnet-4-6	ANTHROPIC_API_KEY=sk-ant-...	★★★★★	★★★★★
OpenAI	gpt-4o	OPENAI_API_KEY=sk-...	★★★★★	★★★★★
xAI Grok	grok-3	XAI_API_KEY=...	★★★★☆	★★★★☆
Azure OpenAI	gpt-4o (enterprise)	AZURE_OPENAI_API_KEY + AZURE_OPENAI_ENDPOINT	★★★★★	★★★★★

Tier Medium — Balanced cost/performance

Provider	Default Model	Setup	JSON	Security
DeepSeek	deepseek-chat	DEEPSEEK_API_KEY=...	★★★★★	★★★★☆
Mistral	mistral-large-latest	MISTRAL_API_KEY=...	★★★★☆	★★★★☆
Cohere	command-r-plus	COHERE_API_KEY=...	★★★★☆	★★★☆☆

Tier Simple — Fast & cheap for parsing

Provider	Default Model	Setup	JSON	Security
Groq	llama-3.3-70b-versatile	GROQ_API_KEY=gsk_...	★★★★☆	★★★★☆
Google Gemini	gemini-2.0-flash	GOOGLE_API_KEY=...	★★★★☆	★★★☆☆
Perplexity	sonar-large-online	PERPLEXITY_API_KEY=...	★★★★☆	★★★☆☆
Together AI	Llama-3-70b-chat-hf	TOGETHER_API_KEY=...	★★★☆☆	★★★☆☆

Tier All — Multi-provider gateway

Provider	Description	Setup
OpenRouter	100+ models via a single key	OPENROUTER_API_KEY=...
9router	Local gateway, 40+ providers	Install: npm i -g 9router && 9router

---

Local / Self-hosted (no API key)

Tool	Auto-detect	Manual setup
Ollama	✅ Auto if running on localhost:11434	ollama serve && ollama pull llama3.2
LM Studio	Via custom	CUSTOM_LLM_BASE_URL=http://localhost:1234/v1
vLLM	Via custom	CUSTOM_LLM_BASE_URL=http://localhost:8000/v1
llama.cpp server	Via custom	CUSTOM_LLM_BASE_URL=http://localhost:8080/v1
LiteLLM proxy	Via custom	CUSTOM_LLM_BASE_URL=http://localhost:4000
Jan.ai	Via custom	CUSTOM_LLM_BASE_URL=http://localhost:1337/v1

Recommended models for local use:

• Reasoning: qwen2.5:72b, deepseek-r1:32b, llama3.1:70b
• Fast: llama3.2:3b, qwen2.5:7b, mistral:7b

---

Custom Endpoint (WorkBuddy, internal API, etc.)

For any endpoint not listed above:

export CUSTOM_LLM_BASE_URL=https://api.workbuddy.ai/v1
export CUSTOM_LLM_API_KEY=<key>
export CUSTOM_LLM_MODEL=<model-name>
export CUSTOM_LLM_NAME=WorkBuddy
export CUSTOM_LLM_TIER=complex

Requirement: the endpoint must be compatible with OpenAI's POST /v1/chat/completions format.

Test compatibility:

curl -X POST $CUSTOM_LLM_BASE_URL/chat/completions \
  -H "Authorization: Bearer $CUSTOM_LLM_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"'$CUSTOM_LLM_MODEL'","messages":[{"role":"user","content":"Reply with: OK"}],"max_tokens":5}'

If the response is: {"choices":[{"message":{"content":"OK"}}]} → compatible ✅

---

Incompatible Models

Model	Reason
Models with non-JSON output	The ReAct loop requires valid JSON on every iteration
Models < 7B parameters	Usually not reliable for complex JSON schemas
Embedding-only models	Cannot generate text
Image-only models	Not relevant for pentesting

---

Recommended Setup

Solo pentester (cost-efficient)

export ANTHROPIC_API_KEY=sk-ant-...   # complex: ReAct loop
export GROQ_API_KEY=gsk_...           # simple: output parsing (free tier)

Team / enterprise

export AZURE_OPENAI_API_KEY=...
export AZURE_OPENAI_ENDPOINT=https://your-resource.openai.azure.com
export ANTHROPIC_API_KEY=...          # fallback

Air-gapped / offline

ollama pull qwen2.5:72b
# PentestAI auto-detects Ollama

WorkBuddy

export CUSTOM_LLM_BASE_URL=https://api.workbuddy.ai/v1
export CUSTOM_LLM_API_KEY=<key>
export CUSTOM_LLM_MODEL=<model>
export CUSTOM_LLM_NAME=WorkBuddy

---

Switching Models Mid-Session

PentestAI> llm                          # view active models
PentestAI> llm add                      # add a new model
PentestAI> llm tier gpt-4o simple       # change tier
PentestAI> llm routing off              # use a single model for everything
PentestAI> llm test                     # test all connections

The mode can be changed at any time without a restart.