Crucible v0.18.0 — Reference Target Suite
We are excited to release Crucible v0.18.0, introducing a built-in Reference Target Suite for ground-truth evaluation, CLI target control, and scoring accuracy calibration.
🚀 What's New?
1. Built-in Reference Target Suite (crucible/targets/)
A library of 12 reference HTTP agents (6 vulnerable + 6 hardened pairs) covering the five most critical agentic security vulnerabilities:
- SQL Injection:
sql_vulnerable/sql_hardened - Shell Command Execution:
shell_vulnerable/shell_hardened - Filesystem Exfiltration:
fs_vulnerable/fs_hardened - MCP Tool Abuse:
mcp_vulnerable/mcp_hardened - Memory Poisoning:
memory_vulnerable/memory_hardened - Privilege Escalation:
delegation_vulnerable/delegation_hardened
Each target operates with zero third-party dependencies (powered purely by standard library http.server and threading), exposing three primary API endpoints:
POST /chat— routes message payloads to evaluate LLM agent behavior.GET /health— exposes vulnerability state and instance attributes.GET /ground_truth— returns structured expected scan outcomes ("pass"/"fail").
2. Target Control CLI Command Group (crucible target)
Manage and validate local target instances directly from your terminal:
# List all 12 reference targets with their expected safety state
crucible target list
# Spin up a specific target (e.g. sql_vulnerable) on a local port
crucible target start --name sql_vulnerable --port 9000
# Start all 12 targets sequentially, verify endpoints, and export a validation report
crucible target validate --output ground_truth_report.json