VGSE: Visually-Grounded Semantic Embeddings for Zero-Shot Learning

The current project page provides pytorch code that implements the following CVPR 2022 paper:
Title: "VGSE: Visually-Grounded Semantic Embeddings for Zero-Shot Learning"
Authors: Wenjia Xu, Yongqin Xian, Jiuniu Wang, Bernt Schiele, Zeynep Akata

Abstract:
Human-annotated attributes serve as powerful semantic embeddings in zero-shot learning. However, their annotation process is labor-intensive and needs expert supervision. Current unsupervised semantic embeddings, i.e., word embeddings, enable knowledge transfer between classes. However, word embeddings do not always reflect visual similarities and result in inferior zero-shot performance. We propose to discover semantic embeddings containing discriminative visual properties for zero-shot learning, without requiring any human annotation. Our model visually divides a set of images from seen classes into clusters of local image regions according to their visual similarity, and further imposes their class discrimination and semantic relatedness. To associate these clusters with previously unseen classes, we use external knowledge, e.g., word embeddings and propose a novel class relation discovery module. Through quantitative and qualitative evaluation, we demonstrate that our model discovers semantic embeddings that model the visual properties of both seen and unseen classes. Furthermore, we demonstrate on three benchmarks that our visually-grounded semantic embeddings further improve performance over word embeddings across various ZSL models by a large margin.

Requirements

Python 3.7.7

PyTorch = 1.8.1

test

Please download the VGSE embeddings, w2v embeddings, and image features here: data, and place it in ./SJE/. The data folder contains the files listed below (take AWA2 dataset as an example):

./data/AWA2/VGSE_SMO.mat: Class embeddings learnt for AWA2 dataset.

./data/AWA2/word2vec_splits.mat: W2v embeddings learnt for AWA2 dataset.

./data/AWA2/res101.mat: Image features.

To reproduce the results of applying our VGSE embeddings on SJE model, please run the following scripts:

sh ./SJE/scripts/AWA2.sh

sh ./SJE/scripts/CUB.sh

sh ./SJE/scripts/SUN.sh

If you feel this repo useful, please cite the following bib entry:

@inproceedings{xu2022vgse,
  author    = {Xu, Wenjia and Xian, Yongqin and Wang, Jiuniu and Schiele, Bernt and Akata, Zeynep},
  title     = {VGSE: Visually-Grounded Semantic Embeddings for Zero-Shot Learning},
  booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year      = {2022}
}

The code is under construction. If you have problems, feel free to reach me at xuwenjia16@mails.ucas.ac.cn

Acknowledgment

We thank the following repos providing helpful components/functions in our work.