Zero-Shot Learning by Harnessing Adversarial Samples
[ACM International Conference on Multimedia 2023]
Zero-Shot Learning by Harnessing Adversarial Samples
Zhi Chen, Pengfei Zhang, Jingjing Li, Sen Wang, Zi (Helen) Huang
Official implementation of the paper "Zero-Shot Learning by Harnessing Adversarial Samples".
Abstract: Zero-Shot Learning(ZSL) aims to recognize unseen classes by generalizing the knowledge, i.e., visual and semantic relationships, obtained from seen classes, where image augmentation techniques are commonly applied to improve the generalization ability of a model. However,this approach can also cause adverse effects on ZSL since the conventional augmentation techniques that solely depend on single-label supervision is not able to maintain semantic information and result in the semantic distortion issue consequently. In other words,image augmentation may falsify the semantic (e.g., attribute) information of an image. To take the advantage of image augmentations while mitigating the semantic distortion issue, we propose a novel ZSL approach by Harnessing Adversarial Samples(HAS). HAS advances ZSL through adversarial training which takes into account three crucial aspects: (1) robust generation by enforcing augmentations to be similar to negative classes, while maintaining correct labels, (2) reliable generation by introducing a latent space constraint to avert significant deviations from the original data manifold, and (3)diverse generation by incorporating attribute-based perturbation by adjusting images according to each semantic attribute’s localization. Through comprehensive experiments on three prominent zero-shot benchmark datasets,we demonstrate the effectiveness of our adversarial samples approach in both ZSL and Generalized Zero-Shot Learning (GZSL) scenarios.
Results reported below show accuracy for base and novel classes for across 11 recognition datasets averaged over 3 seeds.
| Name | CUB T1 | U | S | H | AwA2 T1 | U | S | H | SUN T1 | U | S | H |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| SDGZSL (ICCV21) | 75.5 | 59.9 | 66.4 | 63.0 | 72.1 | 64.6 | 73.6 | 68.8 | 62.4 | 48.2 | 361 | 41.3 |
| HSVA (NeurIPS21) | 62.8 | 52.7 | 58.3 | 55.3 | - | 59.3 | 76.6 | 66.8 | 63.8 | 48.6 | 39.0 | 43.3 |
| APN (NeurIPS20) | 72.0 | 65.3 | 69.3 | 67.2 | 68.4 | 56.5 | 78.0 | 65.5 | 61.6 | 41.9 | 34.0 | 37.6 |
| MSDN (CVPR22) | 76.1 | 68.7 | 67.5 | 68.1 | 70.1 | 62.0 | 74.5 | 67.7 | 65.8 | 52.2 | 34.2 | 41.3 |
| HAS(ours) | 76.5 | 69.6 | 74.1 | 71.8 | 71.4 | 63.1 | 87.3 | 73.3 | 63.2 | 42.8 | 38.9 | 40.8 |
Python 3.6
PyTorch 1.9.1
All experiments are performed with one NVIDIA A6000 GPU.
- Dataset: please download the dataset, i.e., CUB, AWA2, SUN, and change the opt.image_root to the dataset root path on your machine
- Data split and APN image features: please download the data folder and place it in ./data/.
- Pre-trained models: please download the pre-trained models and place it in ./pretrained_models/.
We provide the trained ZSL model for three datasets as below:
| Dataset | ZSL Accuracy | Download link | GZSL Accuracy | Download link |
|---|---|---|---|---|
| CUB | 76.5 | Download | 71.8 | Download |
| AWA2 | 71.4 | Download | 73.3 | Download |
| SUN | 63.2 | Download | 40.8 | Download |
Please refer to the train.sh and eval.sh for detailed instructions on training, evaluating and reproducing the results using our pre-trained models.
If you use our work, please consider citing:
@inproceedings{chen2023zero,
title={Zero-Shot Learning by Harnessing Adversarial Samples},
author={Chen, Zhi and Zhang, Pengfei and Li, Jingjing and Wang, Sen and Huang, Zi},
booktitle={Proceedings of the 31th ACM International Conference on Multimedia (MM'23)},
year={2023}
}If you have any questions, please create an issue on this repository or contact zhi.chen@uq.edu.au.
Our code is based on APN repository. We thank the authors for releasing their code. If you use our model and code, please consider citing the work as well.