aicrypto

Setup

1. First download and extract the CTF challenges from https://huggingface.co/datasets/yuuwwang/aicrypto/blob/main/CTF.tar.gz into the data/ folder. Such that the data/ folder contains data/01-Classic/, data/02-Block/ etc.
2. Build Docker images:
   • In sandbox/: docker build . -t aicrypto-sandbox:latest
   • In server/: docker build . -t aicrypto-sandbox:latest
3. Python dependencies: python3 -m pip install -r requirements.txt

Run the benchmark

Set your API key and run benchmark.py.

Example:

export OPENROUTER_API_KEY="..."
python3 benchmark.py --agent cot --all --model openai/gpt-5.2 --output results_cot.json

Other useful options include --challenge, --category, --parallel, --timeout, --type, and --api-key (see benchmark.py --help).