MemeCLIP: Leveraging CLIP Representations for Multimodal Meme Classification

Siddhant Bikram Shah, Shuvam Shiwakoti, Maheep Chaudhary, Haohan Wang

This is the code repository for our paper MemeCLIP: Leveraging CLIP Representations for Multimodal Meme Classification published in EMNLP 2024.

PrideMM Dataset

The images and labels for the PrideMM dataset are available here (Warning: Insensitive content).

Annotation Terminology

Hate

Class	Terminology
No Hate	0
Hate	1

Targets of Hate

Class	Terminology
Undirected	0
Individual	1
Community	2
Organization	3

Stance

Class	Terminology
Neutral	0
Support	1
Oppose	2

Humor

Class	Terminology
No Humor	0
Humor	1

MemeCLIP Code

All experimental changes can be made through a single file: configs.py.

Directory names can be set in the following variables:

• cfg.root_dir
• cfg.img_folder
• cfg.info_file
• cfg.checkpoint_path
• cfg.checkpoint_file

To train, validate, and test MemeCLIP, set cfg.test_only = False and run main.py.

To test MemeCLIP, set cfg.test_only = True and run main.py.

CSV files are expected to contain image path, text, and label in no particular order.

Pre-trained Weights

Pre-trained weights for MemeCLIP (Hate Classification Task) are available here.

Citation

@inproceedings{shah2024memeclip,
    title = "MemeCLIP: Leveraging CLIP Representations for Multimodal Meme Classification",
    author = "Shah, Siddhant Bikram  and
      Shiwakoti, Shuvam  and
      Chaudhary, Maheep  and
      Wang, Haohan",
    booktitle = "Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing",
    month = nov,
    year = "2024",
    address = "Miami, Florida, USA",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2024.emnlp-main.959/",
    doi = "10.18653/v1/2024.emnlp-main.959",
    pages = "17320--17332",
}

OR

Siddhant Bikram Shah, Shuvam Shiwakoti, Maheep Chaudhary, and Haohan Wang. 2024. MemeCLIP: Leveraging CLIP Representations for Multimodal Meme Classification. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 17320–17332, Miami, Florida, USA. Association for Computational Linguistics.