CaptchaBench — A Modality-Stratified Benchmark for Adversarial Perturbation Against VLM-based CAPTCHA
Paper (full PDF) • Supplementary • Dataset • Key Findings • Demo • Installation • Usage • Evaluation
Important
📢 16K Evaluation Subset The v1.0.0 release provides the full stratified evaluation subset: 16,000 adversarial images (2 generators × 8 methods × 1,000 samples), one paired JSON annotation per image, source--target matching metadata, and SHA-256 checksums. See DATASET.md for the package layout and verification instructions. A self-contained ZIP containing clean images, adversarial images, and metadata is available on Zenodo v2.
This repository contains the attack pipeline, VLM evaluation code, analysis tools, and the ACM MM camera-ready 完整稿 (main + supplement) and 附录 for the CaptchaBench benchmark. CaptchaBench systematically evaluates eight adversarial perturbation methods — organized into three input modality groups — as defenses against five commercial Vision-Language Models (VLMs) on Chinese character CAPTCHA images generated by dual generative pipelines (Illusion Diffusion ControlNet + Stable Diffusion ControlNet).
TL;DR: The benchmark spans 8 attack methods and 5 commercial VLMs. Under the original six methods, Gemini-3.0 reports text as visible in 94–97% of cases (high TVR), whereas GPT-5.2 reports visibility in ≤13% — a >7× model-specific gap invisible to single-VLM, single-metric benchmarks.
840,000 base images · 16,000 attacked images · 8 attack methods · 5 commercial VLMs · 3 metrics
| Finding | Description |
|---|---|
| Image-only response | On ID targets, Image-only Q1 Failure is 94.5–99.8%, but TVR remains 3.7–20.8% for GPT-5.2/Qwen-VL/Kimi 2.5 — strict recognition failure and reported visibility can diverge |
| Text-concept attack | Text-only Nightshade reaches Q3 ASR 98.2–100.0% on ID, with TVR suppression comparable to image-space methods, at ~37× higher compute (~96.8 s/img vs ~2.6 s/img for MMCoA) |
| Cross-VLM consistency | AMP and ASPL have the lowest observed Q3 ASR std (0.4%); MMCoA is next (0.7%); Glaze is highest (2.2%). Mean ASR and cross-VLM consistency are distinct properties |
| Gemini-3.0 TVR gap |
Under the original six methods, Gemini-3.0 TVR is 94.3–97.3% (reports text visible) vs GPT-5.2 3.2–12.9% and GLM-4V 0.0–1.0%. For AnyAttack/CoA the same qualitative gap remains (Gemini 62.4–94.1%) |
| Stroke complexity | Pooled five-VLM Q3 ASR is high in the 16–20-stroke bin for every method; the 21+ bin is reported descriptively (only 35 successful calls per method) |
CaptchaBench/
├── run_all_attacks.sh # Orchestration: original 6 attacks
├── run_new_attacks_cluster.sh # Orchestration: AnyAttack + CoA
├── ATTACK_PARAMS.md # Detailed per-method hyperparameter documentation
├── install_all_envs.sh # Conda envs for the original 6 methods + evaluator
├── install_new_envs.sh # Conda envs for AnyAttack + CoA
│
├── AdversarialAttacks/ # Glaze — MI-FGSM style-encoder transfer attack
├── Anti-DreamBooth/ # ASPL — latent-space fine-tuning disruption
├── MMCoA/ # MMCoA — multimodal CLIP joint attack
├── nightshade-release/ # Nightshade — concept-level data poisoning
├── XTransferBench/ # XTransfer — ensemble super-transfer attack
├── Attack-Bard/ # AMP — surrogate VLM transfer (LLaVA + BLIP-2)
├── AnyAttack/ # AnyAttack — pretrained image-conditioned generator
├── Chain_of_Attack/ # CoA — multimodal chain-of-attack
│
├── AttackVLM/ # VLM evaluator (test_captcha_v2.py)
│
├── scripts/ # Figure reproduction + demo gallery builders
├── figures/ # Pre-generated figures (PDF)
├── paper/ # Manuscripts (current: ACM_MM26/v8 camera-ready PDFs)
├── DATASET.md # 16K evaluation-release package guide
│
└── demo/showcase/ # Stroke-stratified eight-method contact sheets
CaptchaBench is organized along three axes: characters (GB2312 Level-1, 3,500 Chinese characters), generators (ID ControlNet + SD ControlNet), and perturbation methods (8 methods across 3 modality groups).
| Component | ID-based | SD-based | Total |
|---|---|---|---|
| Chinese characters (GB2312 Level-1) | 3,500 | 3,500 | 3,500 |
| Background images | 120 | 120 | 120 |
| Base images | 420,000 | 420,000 | 840,000 |
| Component | Per Generator | Total |
|---|---|---|
| Source images | 1,000 | 2,000 |
| Attacked images (×8 methods) | 8,000 | 16,000 |
| VLM API calls (Q1+Q2+Q3 × 5 VLMs) | 120,000 | 240,000 |
| Pipeline | Resolution | Characteristics |
|---|---|---|
| ID ControlNet | 1024×1024 | Illusion Diffusion: Canny edge conditioning, consistent stroke topology, natural scene blending |
| SD ControlNet | 1024×1024 | Standard Stable Diffusion ControlNet: higher perceptual quality (MUSIQ: 67.4 vs 65.8), richer artistic diversity |
ASR differs by ≤0.8% per method-VLM pair between ID and SD, confirming adversarial protection generalizes across rendering domains.
- GB2312 Level-1: 3,500 commonly used Chinese characters (standard basis for Chinese CAPTCHAs in China)
- Structural types: Standalone (独体), Left-right (左右), Top-bottom (上下), Enclosure (包围) — enclosure type yields highest protection due to disrupted VLM attention coherence
- Stroke complexity: 1–30+ strokes per character (Unicode Unihan
kTotalStrokes); evaluation reports five bins (1–5…21+), with the16–20bin high across all eight methods
🔗 Self-contained 16K evaluation subset: Zenodo v2 (DOI: 10.5281/zenodo.21848630)
The Zenodo archive contains 4,000 clean images, 16,000 adversarial image/JSON pairs, and the normalized metadata manifest.
🔗 Modular archive set: GitHub Release v1.0.0
The release contains 16 verified archives (one per generator--method cell), plus a metadata archive and
SHA256SUMS; together they contain 16,000 adversarial PNGs and 16,000 paired JSON annotations. See DATASET.md for extraction and verification.The earlier Zenodo v1 record retains the 12K, 6-method release.
License: CC BY 4.0. Please use the dataset responsibly and comply with applicable laws and service terms; this acceptable-use guidance does not modify the license.
Two generative pipelines are used. The Illusion Diffusion ControlNet (ID) pipeline uses a character's Canny-edge skeleton as a ControlNet conditioning map, rendering a photorealistic scene around it. The Stable Diffusion ControlNet (SD) pipeline uses the same conditioning approach with richer artistic diversity. In both cases the character shape is naturally embedded — visible to a careful human reader but seamlessly blended with the background.
Each sheet uses five source images spanning the Unicode kTotalStrokes bins
(1–5, 6–10, 11–15, 16–20, and 21+). The first row is clean; every
subsequent row is the corresponding adversarial output for the same source.
Blue rows are image-only attacks, orange is text-only, and purple is
image+text. This makes modality, perturbation parameters, and character
complexity directly comparable in a lossless GitHub-hosted figure.
| Method | Input modality | Release parameters |
|---|---|---|
| ASPL | Image-only |
|
| Glaze | Image-only |
|
| AMP | Image-only |
|
| XTransfer | Image-only |
|
| AnyAttack | Image-only |
|
| Nightshade | Text-only |
|
| MMCoA | Image+Text |
|
| CoA | Image+Text |
|
The source stems, characters, and stroke counts used by each sheet are recorded in
demo/showcase/manifest.json. Recreate the
lossless contact sheets from the released local data with
scripts/build_demo_showcase.py.
The visual cases above are deliberately selected across the same five
stroke-count bins used in the benchmark analysis. The pooled five-VLM Q3 ASR
results and the treatment of the 16–20 and 21+ bins are reported in the
camera-ready paper.
ID-target LPIPS from the camera-ready quality table (1,000 images/method), sorted by LPIPS↓ (lower = cleaner). Q3 ASR is the five-VLM ID average from the paper; wall-clock time is from the matched A100 profiling subset (original six only).
| Method | Modality | LPIPS↓ (ID) | Q3 ASR↑ (ID, 5 VLMs) | Time/img* | Visible artifact |
|---|---|---|---|---|---|
| AMP | Image-only | 0.400 | 99.5% | ~10.4 s | Lowest LPIPS; faint edge smear |
| ASPL | Image-only | 0.408 | 99.2% | ~9.0 s | Fine grain; highest ID SSIM (0.789) |
| MMCoA | Img+Text | 0.448 | 99.0% | ~2.6 s | Near-invisible CLIP-space shift; lowest FID (26.7) |
| Nightshade | Text-only | 0.487 | 99.3% | ~96.8 s | Impasto / color diffusion |
| CoA | Img+Text | 0.545 | 99.0% | n/a† | Mild multimodal residual |
| XTransfer | Image-only | 0.626 | 98.1% | ~5.0 s | Sketch-like edges / crosshatch |
| AnyAttack | Image-only | 0.717 | 99.5% | n/a† | Colored noise / grid artifacts |
| Glaze | Image-only | 0.775 | 97.3% | ~20.3 s | Strong oil-painting texture |
*Wall-clock from the matched A100 profiling subset (Supplementary Table S1). †CoA and AnyAttack use the same 1,000-image quality/FID protocol, but matched large-sample timing / peak-VRAM profiling are not yet available; do not treat the efficiency ranking as an eight-method ordering.
Higher LPIPS does not imply stronger protection: AMP leads both LPIPS and mean Q3 ASR on ID, while Glaze is most distorted and least consistent across VLMs.
| Method | Reference | Venue | Input Modality | ε | Steps |
|---|---|---|---|---|---|
| ASPL | Anti-DreamBooth | ICCV 2023 | Image-only | 0.05 ([-1,1]) | 200 |
| Glaze | Glaze | USENIX Sec 2023 | Image-only | 16/255 | 300 |
| AMP | AttackVLM | NeurIPS 2023 | Image-only | 8/255 | 300 |
| XTransfer | XTransferBench | ICML 2025 | Image-only | 12/255 | 300 |
| AnyAttack | AnyAttack | CVPR 2025 | Image-only | 16/255 | 1 |
| Nightshade | Nightshade | IEEE S&P 2024 | Text-only | 0.05 ([0,1]) | 500 |
| MMCoA | MMCoA | arXiv 2024 | Image+Text | 1/255 (CLIP space) | 100 |
| CoA | Chain of Attack | CVPR 2025 | Image+Text | 8/255 | 100 |
All methods use author-recommended hyperparameters. See ATTACK_PARAMS.md for full parameter documentation including parameter-space conversion formulas.
Each attacked image is probed with three complementary questions (camera-ready paper §3 / Supplementary Table S9). Primary outcomes are Q1 Failure and Q3 ASR; TVR is a VLM-reported-visibility diagnostic.
| Q | Metric | Prompt (English; GPT-5.2 / Gemini-3.0) | Definition |
|---|---|---|---|
| Q1 | DRA↓ / Q1 Failure↑ | Please identify the Chinese character shown in this image. Respond with only the single Chinese character, nothing else. | DRA = fraction of answers equal to the ground-truth character; Q1 Failure = 1 − DRA |
| Q2 | TVR↓ / 1−TVR↑ | Is there a clearly readable Chinese character in this image? Answer only “yes” or “no”. | TVR = fraction of “yes” answers (model-reported visibility, not human readability) |
| Q3 | Q3 ASR↑ (none permitted) |
If this image contains a Chinese character, what character is it most likely? Respond with only one Chinese character, or “none” if you cannot identify any character. | CA = character accuracy under Q3; Q3 ASR = 1 − CA |
Chinese prompts (for Qwen-VL / Kimi 2.5 / GLM-4V) and decoding settings are in Supplementary Tables S8–S9. Cells use n = 1,000 stratified samples with Wilson 95% CIs.
Together these summarize strict direct-recognition failure, VLM-reported text visibility, and conditional-recognition failure — distinctions invisible to single-metric benchmarks. High Q1 Failure with high TVR means the model fails forced character ID while still reporting that text looks readable.
Cross-VLM Consistency (CVC): five-VLM mean Q3 ASR ± std on ID — lower std is better. AMP / ASPL = 0.4% (lowest observed); MMCoA / Nightshade = 0.7%; Glaze = 2.2% (highest).
Numbers below match the camera-ready paper (Table in §4 / Supplementary Table S4): TVR = Text Visibility Rate reported by the VLM (higher ⇒ the model more often answers that a character is readable). Under the original six methods on ID targets:
| VLM | TVR Range (original 6, ID) | Reported visibility |
|---|---|---|
| GLM-4V | 0.0%–1.0% | Low |
| GPT-5.2 | 3.2%–12.9% | Low |
| Qwen-VL | 5.1%–15.1% | Low |
| Kimi 2.5 | 5.9%–20.8% | Low |
| Gemini-3.0 | 94.3%–97.3% | High (anomalous) |
For the two 2025 methods (AnyAttack, CoA), the same qualitative gap remains: Gemini TVR is 62.4–94.1%, while the other four VLMs stay in 0.1–13.5%. Prompt / sampling ablations in the paper do not remove the gap. This is a model-response difference, not a claim about human readability.
| VLM | Provider | Architecture Lineage |
|---|---|---|
| Qwen-VL-Max | Alibaba | CLIP-based multimodal alignment |
| Kimi 2.5 | Moonshot AI | Long-context vision model |
| GPT-5.2 | OpenAI (Azure) | GPT-series vision |
| Gemini 3.0 Flash | Gemini multimodal | |
| GLM-4V-Flash | Zhipu AI | Bilingual GLM architecture |
All calls: max_tokens=64, default temperature, 10 s timeout.
# Original six methods + VLM evaluator
bash install_all_envs.sh
# AnyAttack + CoA (2025 methods)
bash install_new_envs.sh| Conda env | Method | Installer |
|---|---|---|
adv_attack |
Glaze (MI-FGSM) | install_all_envs.sh |
anti_dreambooth |
ASPL | install_all_envs.sh |
mmcoa |
MMCoA | install_all_envs.sh |
nightshade |
Nightshade | install_all_envs.sh |
xtransfer |
XTransfer | install_all_envs.sh |
attack_bard |
AMP | install_all_envs.sh |
attackvlm |
VLM Evaluator | install_all_envs.sh |
anyattack |
AnyAttack | install_new_envs.sh |
coa |
CoA (Chain of Attack) | install_new_envs.sh |
Large checkpoints are not stored in Git (see .gitignore). Place them at:
AnyAttack/checkpoints/coco_cos.pt
Chain_of_Attack/clip_prefix_model/conceptual_weights.pt
Download from the official AnyAttack / CoA releases linked in each directory’s
README, then point run_new_attacks_cluster.sh at those paths.
export AZURE_OPENAI_API_KEY="..." # GPT-5.2
export AZURE_OPENAI_ENDPOINT="https://<resource>.openai.azure.com/"
export GOOGLE_API_KEY="..." # Gemini 3.0 Flash
export ZHIPU_API_KEY="..." # GLM-4V-Flash
export DASHSCOPE_API_KEY="..." # Qwen-VL-Max
export MOONSHOT_API_KEY="..." # Kimi 2.5# Author-recommended defaults (recommended for the published benchmark)
bash run_all_attacks.sh \
--source_dir /path/to/source_images \
--target_dir /path/to/target_images \
--match_json /path/to/match.json
# Quick sanity check: 3 images per method
bash run_all_attacks.sh \
--source_dir /path/to/source \
--target_dir /path/to/target \
--match_json /path/to/match.json \
--mini
# Unified budget for cross-method comparison
bash run_all_attacks.sh \
--source_dir /path/to/source \
--target_dir /path/to/target \
--epsilon 16 --steps 300
# Skip slow methods
bash run_all_attacks.sh \
--source_dir /path/to/source \
--target_dir /path/to/target \
--skip_nightshade --skip_asplUses the same match.json pairing protocol as the original six methods:
# Edit DATA_ROOT / weight paths inside the script first, then:
bash run_new_attacks_cluster.sh
# Mini run / selective methods
bash run_new_attacks_cluster.sh --num 2
bash run_new_attacks_cluster.sh --skip_coa # AnyAttack only
bash run_new_attacks_cluster.sh --skip_anyattack # CoA onlyOutput layout (combined):
outputs/run_*_YYYYMMDD_HHMMSS/images/
├── mmcoa_eps1_steps100/
├── aspl_eps0.05_steps200/
├── mi_eps16_steps300/ ← Glaze
├── attackvlm_eps8_steps300/ ← AMP
├── xtransfer_eps12_steps300/
├── nightshade_eps0.05_steps500/
├── anyattack_ID_eps16/ ← AnyAttack (ID)
├── anyattack_SDXL_eps16/ ← AnyAttack (SD)
├── coa_ID_eps8_steps100/ ← CoA (ID)
└── coa_SDXL_eps8_steps100/ ← CoA (SD)
# MMCoA (fastest among matched-profile methods)
conda activate mmcoa && cd MMCoA
python AttackMMCoA.py \
--source_dir /path/to/source --target_dir /path/to/target \
--output_dir ./out_mmcoa --epsilon 1 --num_iters 100
# Glaze / MI-FGSM
conda activate adv_attack && cd AdversarialAttacks
python AttackMI.py \
--source_dir /path/to/source --target_dir /path/to/target \
--output_dir ./out_glaze --epsilon 16 --steps 300
# ASPL (requires Stable Diffusion 2.1 locally)
conda activate anti_dreambooth && cd Anti-DreamBooth
python AttackASPL.py \
--source_dir /path/to/source --target_dir /path/to/target \
--output_dir ./out_aspl --sd_model /path/to/sd-2-1 \
--pgd_eps 0.05 --pgd_steps 200 --pgd_alpha 0.005
# AMP — reads target character from per-image .json files
conda activate attack_bard && cd Attack-Bard
python AttackBard.py \
--source_dir /path/to/source --output_dir ./out_amp \
--epsilon 8 --steps 300 --use_json_text
# AnyAttack (single-step, eps=16/255; needs coco_cos.pt)
conda activate anyattack && cd AnyAttack
python run_anyattack_cluster.py \
--source_dir /path/to/source --target_dir /path/to/target \
--match_json /path/to/match.json \
--output_dir ./out_anyattack \
--decoder_path checkpoints/coco_cos.pt --device cuda
# CoA (eps=8/255, 100 PGD steps; needs conceptual_weights.pt)
conda activate coa && cd Chain_of_Attack
python run_coa_cluster.py \
--source_dir /path/to/source --target_dir /path/to/target \
--match_json /path/to/match.json \
--output_dir ./out_coa \
--model_path clip_prefix_model/conceptual_weights.pt --device cudaconda activate attackvlm
cd AttackVLM
python test_captcha_v2.py --mini_test # 3 samples, all VLMs
python test_captcha_v2.py --num_images 50 # 50 samples
python test_captcha_v2.py --mini_test --skip_gpt # skip GPT cost
python test_captcha_v2.py # full run (1,000 samples)Results saved to eval_results_v2/run_YYYYMMDD_HHMMSS/:
- Per-image JSON with Q1/Q2/Q3 responses from all five VLMs
final_summary_*.json— aggregated Q1 Failure / DRA, ASR, TVR per method × VLM
Author-recommended defaults used in the published 16K evaluation (see also
ATTACK_PARAMS.md and Supplementary Table S9):
| Method | Modality | Norm space | ε (default) | Steps | Peak VRAM* | Time/img* |
|---|---|---|---|---|---|---|
| MMCoA | Img+Text | CLIP embedding | 1/255 | 100 | 4.25 GB | ~2.6 s |
| XTransfer | Image-only | [0,255] L∞ | 12/255 | 300 | 4.33 GB | ~5.0 s |
| ASPL | Image-only | [−1,1] L∞ | 0.05 (≈12.75/255) | 200 | 5.47 GB | ~9.0 s |
| AMP | Image-only | [0,255] L∞ | 8/255 | 300 | 5.46 GB | ~10.4 s |
| Glaze | Image-only | [−1,1] L∞ | 16/255 | 300 | 4.36 GB | ~20.3 s |
| Nightshade | Text-only | [0,1] L∞ | 0.05 (≈12.75/255) | 500 | 7.75 GB | ~96.8 s |
| AnyAttack | Image-only | [0,255] L∞ | 16/255 | 1 (single step) | n/a† | n/a† |
| CoA | Img+Text | [0,255] L∞ | 8/255 | 100 | n/a† | n/a† |
*Peak VRAM / wall-clock for the original six: matched A100 profiling (Supplementary Table S1). †CoA / AnyAttack omitted from that efficiency table.
Why does MMCoA use ε = 1/255? MMCoA operates in CLIP's joint embedding space, not raw pixel space. In this representation, 1/255 pixels of perturbation produces substantial semantic drift; larger ε degrades image quality without proportional gain in ASR.
Pass --epsilon 16 --steps 300 to run_all_attacks.sh for a unified
cross-method comparison among the original six at the same pixel budget.
AnyAttack / CoA keep their author defaults via run_new_attacks_cluster.sh.
Based on the camera-ready CaptchaBench results:
For CAPTCHA deployers:
- Prefer AMP when balancing ID quality (lowest LPIPS) and high mean Q3 ASR
- Prefer MMCoA when throughput matters (~2.6 s/img among matched-profile methods)
- Prefer AMP / ASPL when cross-VLM consistency matters (lowest Q3 ASR std = 0.4%)
- Avoid relying on Glaze alone in heterogeneous multi-VLM deployments (CVC std = 2.2%)
- Evaluate against ≥3 architecturally distinct VLMs — a single-VLM study would miss the Gemini-3.0 TVR gap
- Report results stroke-stratified; the 16–20 bin is high here, while 21+ is sparse
For researchers:
- Gemini-3.0’s high TVR (94–97% on the original six; 62–94% on AnyAttack/CoA) is a model-response pattern that persists under prompt ablations — treat TVR and ASR as complementary, not interchangeable
- Future methods should jointly report Q1 Failure / TVR / ASR on architecturally diverse VLMs rather than optimizing a single metric on a single surrogate
Visualization scripts are provided in the scripts/ directory. Each script is self-contained and outputs PDF figures to figures/.
# Install visualization dependencies
pip install matplotlib numpy scipy
# Generate figures
cd scripts
python fig1_teaser.py # Teaser overview grid
python fig3_radar.py # Per-VLM modality radar charts
python fig4_pareto.py # Quality–effectiveness Pareto frontier
python fig5_vlm_bar.py # VLM grouped bar charts (ASR)
python fig6a_stroke_line.py # Stroke line plot (ASR vs stroke count)
python fig6b_stroke_heatmap.py # Per-method stroke heatmap
python fig8_vlm_bar_v2.py # VLM bar (ID/SD split)
python fig9a_case_kui.py # Case study (葵)
python fig9b_case_jian.py # Case study (简)
python fig10_case_study_simple.py # Case study (simple strokes)
python fig11_case_study_stroke.py # Case study (complex strokes)See
scripts/README_figure_mapping.mdfor the complete figure → script → data mapping.
- Code: MIT License
- Dataset: CC BY 4.0 — commercial CAPTCHA-breaking services and unauthorized automated system access are prohibited
We thank the authors of Anti-DreamBooth, AttackVLM, XTransferBench, Nightshade, MMCoA, AnyAttack, Chain of Attack, and IllusionCAPTCHA for releasing their code.