Configurable AI Security Testbed - a deliberately vulnerable platform for developing and measuring LLM attack methodology.
Think DVWA, but for LLMs. You get an intentionally vulnerable application, a configurable defence stack, and real-time telemetry showing exactly why each message was blocked or passed.
I rapidly came to the conclusion that the security testing I was performing against enterprise LLM implementations was largely "blind" in that I had to infer backend technology, ellicit error messages to fingerprint, and spend much time coercing tool calls and needed a way to develop my primitives in a controlled environment that afforded me the insight as to what was happening and where.
If you build AI systems, CAST shows you where your defences fail and why. If you break them, it gives you a controlled environment to develop methodology and test techniques against varying levels of defence.
- Domain-specific vulnerable applications - healthcare, banking, property - each with intentional misconfigurations mapped to real-world vulnerability classes, with more to be added as time goes on.
- Configurable defence stack - syntactic filters, semantic similarity, LLM-as-judge, Azure Content Safety, OpenAI Moderation. Three presets (easy/medium/hard and Insane planned for later) or build your own.
- Real-time Autopsy panel - defence verdicts, semantic scores, LLM judge reasoning, raw tool call parameters, and rendered output for every turn
Vibecoded🌟AgeNtiCAlly ENginEERed 🌟 Front and Backend... which will most likely have bugs that creep through my alpha testing, these will be identified and squished as time goes on, please be patient
- API keys, you will need to supply these yourself. Currently CAST will support:
- Azure Content Safety
- Anthropic
- OpenAI
As a starting point, you can get away with JUST an OpenAI key, this will power your primary chatbot instance and LLM-as-Judge
This tool will have regular blog iterations that cover different attack chains and techniques
🚧 Open source release in progress. Documentation and setup guides are being finalised. Watch this repo for updates.