ToViLaG

Official script of EMNLP 2023 paper: ToViLaG: Your Visual-Language Generative Model is Also An Evildoer.

Metrics

WInToRe Metric

Run the following command to compute the WInToRe metric.

python metrics/toxicity/wintore.py --input wintore_input.txt --output wintore_output.txt --start 0 --end 1 --M 20

Arguments include:

• --input: The file for the input toxicity list. See wintore_input.txt for an example.

• --output: The file for the output toxicity list. See wintore_output.txt for an example.

• --start: Start of the threshold

• --end: End of the threshold

• --M: The number of the threshold set.

Quality Metrics

Image-to-text metrics: BERTScore, ROUGE, and CLIPSIM.

Text-to-image metrics: IS, FID, and CLIPSIM.

Toxicity Classifier

Text toxicity classifier: Perspective API. A simple direct implementation is available here.

Image toxicity classifiers: We use part of toxic images to fine-tune three ViT-Huge models for the three types of toxicity, respectively.

ToViLaG Dataset

Statistic

Category	Number of Image	Number of Text
Mono-toxic pairs <toxic image, non-toxic text>	4,349	10,000
Mono-toxic pairs <toxic text, non-toxic image>	10,000	9,794
Co-toxic pairs <toxic text, toxic image>	5,142	9,869
Provocative text prompts		902
Unpaired	21,559	31,674

Unpaired data

Unpaired toxic images:

• Pornographic images: Download the NSFW Image Classification dataset from Kaggle. We use the porn class in the test set for toxicity benchmarking, with a total of 8,595 images.
• Violent images: Request UCLA Protest Image Dataset from here provided in Won et. al., Protest Activity Detection and Perceived Violence Estimation from Social Media Images, ACM Multimedia 2017. We use the combination of the protest class from the train and test sets for toxicity benchmarking, with a total of 11,659 images.
• Bloody images: Please contact me via email to obtain the images, totaling 1,305 images for toxicity benchmarking.

Unpaired toxic text: We use part of them (21,805 text) for toxicity benchmarking, which can be downloaded from here;

Mono-toxic pairs

<toxic image, non-toxic text>

• Toxic images: Same with the unpaired toxic images.
• Non-toxic text: Generated by GIT for toxic images. Filtered by PerspectiveAPI, PPL, CLIPScore, Jaccard similarity.

<toxic text, non-toxic image>

• Ready-made：Detected and collected from existing VL datasets.

  Datasets	Number of toxic pairs
  COCO	570
  Flickr30k	233
  CC12M	4286

• Augmented：

  • Non-toxic images: From part of COCO.
  • Toxic text: Rewritten by fBERT on corresponding text of non-toxic images; Filtered by PerspectiveAPI, PPL, CLIPScore, Jaccard similarity.

Co-toxic pairs

• Toxic images: Same with the unpaired toxic images.

• Toxic text: Generated by BLIP for toxic images; Filtered by PerspectiveAPI, PPL, CLIPScore, Jaccard similarity.

Innocuous provocative text prompts

Constructed by a gradient-guided search method on Stable Diffusion.

Download the prompts from here.

Toxicity Analysis

Toxicity Benchmarking

Image-to-text generation

We use 21,559 toxic images to evaluate the I2T models.

All models apply the top-k and top-p sampling to generate outputs in our paper. The toxicity evaluation results of each model are as follows:

Models	TP% ↑	WInToRe% ↓
OFA	3.41	90.16
VinVL	2.06	89.56
CLIP-ViL$_{RN50}$	0.74	88.99
GIT	11.57	86.13
GRIT	12.79	84.70
LLaVA	29.25	80.89
BLIP	32.51	75.66
BLIP2$_{OPT2.7B-COCO}$	37.61	66.55
BLIP2$_{OPT2.7B}$	40.41	64.76

Text-to-image generation

We use 21,805 toxic prompts and 902 provocative prompts to evaluate the T2I models.

The toxicity evaluation results of each model are as follows:

Models	Toxic Prompts		Provocative Prompts
	TP% ↑	WInToRe% ↓	TP% ↑	WInToRe% ↓
CogView2	8.10	81.37	44.68	-8.59
DALLE-Mage	10.19	80.96	33.15	-7.29
OFA	19.08	80.64	37.03	-7.44
Stable Diffusion	23.32	80.12	100	-19.02
LAFITE	21.48	79.33	27.38	-6.51
CLIP-GEN	22.93	79.97	7.32	1.18

Toxicity Injection

We use the mono-toxic pairs and the co-toxic pairs to fine-tune each model, respectively.

Image-to-text generation models: GIT, GRIT, BLIP

Text-to-image generation models: Stable Diffusion, LAFITE, CLIP-GEN

SMIB Detoxification Method

We apply the SMIB method into three models in our paper: GIT, GRIT, and BLIP.

We use 5,000 non-toxic image-text pairs from COCO and 5,000 toxic ones from our co-toxic pairs for training. We take the implementation of BLIP with SMIB as an example.

Run the following command to train the detoxification process of the BLIP model:

python method/BLIP/train_caption_detox.py --output_dir outputs/detox --device 1

Infer the detoxified text for toxic images:

python method/BLIP/inference.py --image_path /path/to/toxic_images/ --model_size large --device 1

Contact

If you have any problems on implementation or any other questions, feel free to post a issue or email me (wangxinpeng@tongji.edu.cn).