Title: Triple Alignment Strategies for Zero-shot Phrase Grounding under Weak Supervision
Authors: Pengyue Lin, Ruifan Li, Yuzhe Ji, Zhihan Yu, Fangxiang Feng, Zhanyu Ma, Xiaojie Wang
Affiliation: Beijing University of Posts and Telecommunications
Conference: ACM Multimedia 2024 (Full Paper, pp. 4312–4321)
Phrase grounding (PG) aims to localize objects referred to by noun phrases in an image. Recent works have explored either weakly supervised PG (without region-level annotations) or zero-shot PG (generalizing from seen to unseen categories). However, both settings have limitations in real-world applications due to scarce annotations and limited category coverage during training.
In this paper, we propose WS-ZSG, a novel framework for Zero-shot Phrase Grounding under Weak Supervision. Our approach is built upon three key alignment strategies:
- Region-Text Alignment (RTA): Leverages CLIP to establish region-level attribute associations between image regions and text descriptions.
- Domain Alignment (DomA): Minimizes the distributional gap between seen classes in training data and those from pre-training domains.
- Category Alignment (CatA): Integrates category semantics and region-category relationships to enhance generalization to unseen categories.
Extensive experiments demonstrate that our method outperforms existing zero-shot approaches and achieves competitive performance compared to fully and weakly supervised methods. The code and data will be made publicly available after the double-blind review phase.
If you find this work helpful, please cite our paper:
@inproceedings{lin2024triple,
title={Triple alignment strategies for zero-shot phrase grounding under weak supervision},
author={Lin, Pengyue and Li, Ruifan and Ji, Yuzhe and Yu, Zhihan and Feng, Fangxiang and Ma, Zhanyu and Wang, Xiaojie},
booktitle={Proceedings of the 32nd ACM International Conference on Multimedia},
pages={4312--4321},
year={2024}
}This project is released under the MIT License.