redcell

An adversarial test harness for LLM apps and MCP servers — points a battery of real attacks at a target and reports which ones actually got through, organized by the OWASP Top 10 for LLM Applications rather than an invented taxonomy.

redcell scan --target-kind mcp --command python --args "-m your_server"
redcell scan --target-kind chat --anthropic-key $ANTHROPIC_API_KEY

Why this exists

Most people building agent tools right now aren't testing them this way. Coding-agent security disclosures (Manifold Security's report of eight flaws across seven command-line AI agents, mid-2026) made it clear this gap is real, not hypothetical. This project also isn't a stretch from scratch — it formalizes instincts I'd already been exercising by hand: spotting a prompt-injection pattern packaged as an "agent prompt" in a cold email, and finding real security-relevant bugs in other people's PRs (chroma-core, weaviate, qdrant). This turns that kind of scrutiny into something repeatable.

How it's built

target (chat LLM or MCP server)
        │
        ▼
   probe registry ──┬── LLM01 Prompt Injection (direct + indirect)
                     ├── LLM02 Insecure Output Handling
                     ├── LLM06 Sensitive Information Disclosure
                     └── LLM08 Excessive Agency (schema-driven MCP fuzzing)
        │
        ▼
   evaluator ──┬── heuristic (fast, deterministic)
               └── LLM-as-judge (for genuinely ambiguous cases)
        │
        ▼
   ScanReport → JSON (CI) or Markdown (human)

Two target kinds, on purpose, not one forced abstraction. A chat LLM and an MCP server's tool surface are fundamentally different interaction shapes — one is a conversation, the other a set of typed function calls — so ChatTarget and MCPTarget are separate interfaces. Probes declare which kind they need (target_kind = "chat" | "mcp"), and the runner only points them at a matching target.

The MCP probes are schema-driven, not hardcoded. Every MCP server exposes different tools with different argument names, so PathTraversalProbe and CommandInjectionProbe inspect a target's actual list_tools() output, heuristically pick out string parameters that look path- or command-like (by name/description), and fire a payload at each one they find. A server with no such parameters — like this repo's own critique-mcp, whose tools take a GitHub owner/repo/pr_number rather than a local path — has nothing for these probes to fire at, and running redcell against it confirms exactly that (0 applicable attempts, not a false alarm).

Two real bugs found while building this, and why the fix matters

A false negative in the first version of the path-traversal probe. It fired OS-specific payloads (/etc/passwd, ..\Windows\win.ini) and judged the result by scanning the response for words like "error" or "not found." That looked reasonable until it was run against a genuinely vulnerable fixture tool on Windows: /etc/passwd doesn't exist there, so the naive tool's open() call failed with a normal file-not-found error — which the keyword scan then read as "properly rejected." The same keyword-based check would have called a completely unvalidated tool "safe," for a reason that had nothing to do with validation. The fix: stop guessing from error text entirely. The probe now creates a real, uniquely-named canary file and asks the tool to read it by absolute path; the canary's exact content coming back is unambiguous proof, not an inference. (CommandInjectionProbe uses the same trick — an echoed random marker — for the same reason.)

Conflating "the probe crashed" with "the target is vulnerable." Running an early build against a real external MCP server that happened to be unreachable produced a report with two "CRITICAL" and "HIGH" findings — except the "findings" were just the probe failing to connect, not the target failing to defend itself. A report that can't tell "we found a vulnerability" apart from "we couldn't finish the test" isn't trustworthy for the one thing a report like this needs to be trustworthy for. ProbeResult now has a separate errored flag; ScanReport.findings excludes errored attempts unconditionally, and the Markdown report prints them in their own "Could not complete" section instead.

Setup

python -m venv .venv && source .venv/bin/activate   # Windows: .venv\Scripts\activate
pip install -r requirements.txt

Usage

Against an MCP server:

redcell scan --target-kind mcp --command python --args "-m critique_mcp.server" --cwd /path/to/critique-mcp

Against a raw Claude API endpoint:

redcell scan --target-kind chat --anthropic-key $ANTHROPIC_API_KEY --model claude-sonnet-4-5

Against your own app (a chatbot, an agent loop — anything with a function that takes messages and returns text): wrap it with CallableChatTarget(your_function) and call run_scan() directly from Python instead of the CLI.

List what's registered:

redcell list-probes

Exit code is 1 if any findings were produced (so CI can gate on it), 0 otherwise. Output defaults to Markdown; pass --output json for machine consumption.

Honest scope

• Covers 4 of the OWASP LLM Top 10 (LLM01, LLM02, LLM06, LLM08) — the ones that are actually testable by sending requests to a black-box target. The rest (e.g. LLM04 Data and Model Poisoning) need access to training data or the model weights themselves, which this tool never has.
• The excessive-agency probes report "this tool accepted a value that should have been scoped" — a real, verified fact — not "this is definitely exploitable in your specific deployment." Context matters; treat every finding as a strong lead to a human review, not a CVE.
• The LLM-judge evaluator makes a real API call and costs real (small) money per probe that uses it. Heuristic evaluators are free and used wherever a substring/keyword check is actually sufficient.

Tests

pytest

Every probe is tested against a scripted fake target (deterministic, no network, no API key needed). The MCP target's actual wire-protocol behavior is additionally tested against a real MCP server fixture over stdio, not a mock of the protocol — see tests/fixtures/fake_mcp_server.py.