VLM Attack Tutorial: ViT Training and Attacks

本项目是一个面向 VLM 安全入门的本地化实验仓库。目前只覆盖 ViT 这一部分：先训练一个本地 ViT 图像分类器，再在这个分类器上做白盒攻击和注意力区域攻击。后续会继续补黑盒攻击、迁移攻击，以及更贴近 VLM 行为的视觉输入攻击。

This repository is a local, staged tutorial project for VLM safety. The current scope is the ViT section: train a local ViT image classifier, then run white-box and attention-guided attacks against it. Black-box attacks, transfer attacks, and more VLM-specific visual-input attacks are planned next.

教程目录 / Tutorial Posts

Part	中文	English
1	ViT 理论与训练流程	ViT Theory and Training Flow
2	ViT 到 VLM 安全：白盒攻击与注意力区域攻击	From ViT to VLM Safety: White-Box and Attention-Region Attacks

博客正文不放进仓库提交历史；仓库只保留可复现代码、依赖配置和项目说明。

Blog drafts are intentionally not tracked here. The repository keeps reproducible code, dependency files, and project documentation only.

当前流程 / Current Workflow

Stage	中文说明	English Description	Code	Main Outputs
0	准备 uv 环境和数据目录	Set up the uv environment and data directory	pyproject.toml, uv.lock, requirements*.txt	.venv/, data/
1	微调 ViT-Base，得到本地分类器	Fine-tune ViT-Base and save a local classifier	scripts/vit_pet_finetune.py	artifacts/vit_pet_full/
2	运行 FGSM、PGD 和可见 patch 攻击	Run FGSM, PGD, and visible patch attacks	scripts/vit_pet_attacks.py	artifacts/vit_pet_attacks/
3	运行注意力区域 PGD	Run attention-guided region PGD	scripts/vit_attention_region_attacks.py	artifacts/vit_attention_attacks/
4	黑盒和更多 VLM 攻击方法	Black-box and broader VLM attacks	planned	planned

目录管理 / Directory Policy

.
├── scripts/
│   ├── vit_pet_finetune.py              # ViT fine-tuning on Oxford-IIIT Pet
│   ├── vit_pet_attacks.py               # FGSM, PGD, and visible patch attacks
│   └── vit_attention_region_attacks.py  # attention visualization and masked region PGD
├── pyproject.toml                       # uv project config
├── uv.lock                              # pinned uv lockfile
├── requirements.txt                     # fallback dependency list
├── requirements-torch-cu124.txt         # fallback CUDA 12.4 PyTorch wheels
└── README.md

本地生成但不提交的目录：

• .venv/: local uv environment
• data/: TorchVision-downloaded Oxford-IIIT Pet dataset
• artifacts/: metrics, figures, learned patches, checkpoints
• blog/: local blog drafts
• *.log, *.pth, *.pt, *.npy: run logs and generated model/data artifacts

Generated local directories are ignored by Git:

• .venv/: local uv environment
• data/: TorchVision-downloaded Oxford-IIIT Pet dataset
• artifacts/: metrics, figures, learned patches, checkpoints
• blog/: local blog drafts
• *.log, *.pth, *.pt, *.npy: run logs and generated model/data artifacts

0. 环境准备 / Environment

推荐使用 uv sync：

Recommended setup:

uv sync

如果需要手动创建环境：

Manual setup:

uv venv .venv --python python3.12
source .venv/bin/activate
uv pip install -r requirements-torch-cu124.txt
uv pip install -r requirements.txt

PyTorch 和 TorchVision 在 pyproject.toml / uv.lock 中固定为 CUDA 12.4 wheels。运行前建议先用 nvidia-smi 选择空闲 GPU。

PyTorch and TorchVision are pinned to CUDA 12.4 wheels through pyproject.toml / uv.lock. Check nvidia-smi first and choose an idle GPU before running long jobs.

1. 训练 ViT 分类器 / Train the ViT Classifier

快速 smoke run：

Quick smoke run:

CUDA_VISIBLE_DEVICES=<GPU_ID> uv run python scripts/vit_pet_finetune.py \
  --epochs 1 \
  --max-train-samples 256 \
  --max-val-samples 256 \
  --batch-size 32 \
  --output-dir artifacts/vit_pet_smoke \
  --save-model

完整复现 run：

Full reproduction run:

CUDA_VISIBLE_DEVICES=<GPU_ID> uv run python scripts/vit_pet_finetune.py \
  --epochs 50 \
  --batch-size 32 \
  --num-workers 4 \
  --output-dir artifacts/vit_pet_full \
  --save-model

训练脚本会自动通过 TorchVision 下载 Oxford-IIIT Pet 到 data/。默认路径冻结 ViT backbone，只训练分类头。

The training script downloads Oxford-IIIT Pet through TorchVision into data/. By default, it freezes the ViT backbone and trains only the classifier head.

主要输出 / Main outputs:

• artifacts/vit_pet_full/metrics.json
• artifacts/vit_pet_full/training_curves.png
• artifacts/vit_pet_full/confusion_matrix.png
• artifacts/vit_pet_full/confusion_matrix.npy
• artifacts/vit_pet_full/vit_finetuned_final.pth

2. FGSM、PGD 和 Patch 攻击 / FGSM, PGD, and Patch Attacks

训练完成后，使用保存的本地 checkpoint 运行攻击：

After training, run attacks with the saved local checkpoint:

CUDA_VISIBLE_DEVICES=<GPU_ID> uv run python scripts/vit_pet_attacks.py \
  --checkpoint artifacts/vit_pet_full/vit_finetuned_final.pth \
  --max-val-samples 512 \
  --patch-train-samples 512 \
  --patch-train-epochs 8 \
  --patch-size 96 \
  --batch-size 32 \
  --num-workers 4 \
  --output-dir artifacts/vit_pet_attacks \
  --num-examples 8

如果只想跑 FGSM 和 PGD，可以加 --skip-patch。

Use --skip-patch if you only want FGSM and PGD.

主要输出 / Main outputs:

• artifacts/vit_pet_attacks/attack_metrics.json
• artifacts/vit_pet_attacks/attack_examples.png
• artifacts/vit_pet_attacks/learned_patch.png
• artifacts/vit_pet_attacks/learned_patch.pt

3. 注意力区域攻击 / Attention-Guided Region Attacks

该脚本会读取 class-token attention，选择高注意力区域，然后只在该区域内运行 masked PGD。

This script reads class-token attention, selects a high-attention region, and runs masked PGD only inside that region.

CUDA_VISIBLE_DEVICES=<GPU_ID> uv run python scripts/vit_attention_region_attacks.py \
  --checkpoint artifacts/vit_pet_full/vit_finetuned_final.pth \
  --max-val-samples 256 \
  --batch-size 16 \
  --num-workers 4 \
  --global-eps 0.03137254901960784 \
  --semantic-eps 0.00784313725490196 \
  --region-eps 0.06274509803921569 \
  --pgd-alpha 0.00392156862745098 \
  --pgd-steps 20 \
  --region-size 96 \
  --num-examples 6 \
  --output-dir artifacts/vit_attention_attacks

主要输出 / Main outputs:

• artifacts/vit_attention_attacks/attention_attack_metrics.json
• artifacts/vit_attention_attacks/attention_attack_examples.png

如果只允许使用本地 Hugging Face 缓存，可以加 --local-files-only。

Use --local-files-only to force model loading from the local Hugging Face cache.

TODO / Roadmap

• 用 uv 本地复现 ViT fine-tuning。

• 添加 FGSM 和 PGD 白盒攻击。

• 添加 targeted visible patch attack。

• 添加 attention-guided local region PGD 和注意力可视化。

• 添加无梯度场景下的黑盒攻击。

• 添加 query-based 和 transfer-based attack baseline。

• 添加更贴近 VLM 的攻击方法，例如误导性视觉上下文、图中文字干扰、UI screenshot manipulation、image-prompt interaction cases。

• 添加超出 top-1 classification accuracy 的 VLM 输出评测脚本。

• 添加防御和诊断笔记，例如 attention inspection、input transformations、robustness checks。

• Reproduce ViT fine-tuning locally with uv.

• Add FGSM and PGD white-box attacks.

• Add targeted visible patch attack.

• Add attention-guided local region PGD and attention visualizations.

• Add black-box attacks where gradients are unavailable.

• Add query-based and transfer-based attack baselines.

• Add more VLM-specific attacks, such as misleading visual context, OCR/text-in-image interference, UI screenshot manipulation, and image-prompt interaction cases.

• Add VLM-output evaluation beyond top-1 classification accuracy.

• Add defense and diagnostic notes, including attention inspection, input transformations, and robustness checks.

Notes

中文：当前代码只用于本地鲁棒性分析和 ViT/VLM 安全入门实验。仓库不包含预训练模型权重或训练出的 checkpoint；请先在本地训练分类器，再运行攻击脚本。

English: The current code is intended for local robustness analysis and ViT/VLM safety study. The repository does not include pretrained weights or generated checkpoints; train the classifier locally before running the attack scripts.