CLIP-based Adaptive Graph Attention Network for Large-Scale Unsupervised Multi-modal Hashing Retrieval

This repository contains the author's implementation in PyTorch for the paper "CLIP-based Adaptive Graph Attention Network for Large-Scale Unsupervised Multi-modal Hashing Retrieval".

Introduction

Unsupervised cross-modal hashing retrieval has been extensively studied due to its advantages in storage, retrieval efficiency, and label independence. However, there are still two obstacles to existing unsupervised methods: (1) Existing unsupervised methods suffer from inaccurate similarity as simple features do not describe fine-grained multimodal relationships. (2) Existing methods suffer from unbalanced multimodal learning due to the different coding capabilities of different modal networks. To address these obstacles, we devised an effectiveCLIP-based Adaptive Graph Attention Network (CAGAN) for Large-Scale Unsupervised Multi-modal Hashing Retrieval. Firstly, we use the multimodal transformer model CLIP to extract cross-modal fine-grained features and exploit multiple data similarities to mine similar information from different perspectives in multimodal data and perform similarity enhancement. In addition, we present an adaptive graph attention hashing module to assist in generating hash codes, which uses an attention mechanism to learn relation-based similarity from image-text modality. It aggregates the essential neighborhood message of neighboring data nodes through the graph neural networks to generate more discriminative hash codes. Sufficient experiments on three benchmark datasets demonstrate that the proposed AGAH outperforms existing advanced unsupervised cross-modal hashing methods.

---

Dependencies

Please, install the following packages:

• Python (>=3.8)
• pytorch
• torchvision
• h5py
• CLIP

Datasets

You can download the features of the datasets from: For datasets, we follow Deep Cross-Modal Hashing's Github (Jiang, CVPR 2017). You can download these datasets from:

• Wikipedia articles, Link
• MIRFLICKR25K, [OneDrive], [Baidu Pan, password: 8dub]
• NUS-WIDE (top-10 concept), [OneDrive], [Baidu Pan, password: ml4y]
• MS-COCO, BaiduPan(password: 5uvp)

Implementation

Here we provide the implementation of our proposed models, along with datasets. The repository is organized as follows:

• data/ contains the necessary dataset files for NUS-WIDE, MIRFlickr, and MS-COCO;
• models.py contains the implementation of the model;

Finally, main.py puts all of the above together and can be used to execute a full training run on MIRFlcikr, NUS-WIDE, or MS-COCO.

Process

• Place the datasets in data/
• Set the experiment parameters in main.py.
• Train a model:

python main.py

• Modify the parameter EVAL = True in main.py for evaluation:

python main.py

Citation

If you find our work or the code useful, please consider cite our paper using: