Fine-Grained Zero-Shot Learning: Advances, Challenges, and Prospects

Recent zero-shot learning (ZSL) approaches have integrated fine-grained analysis, i.e.,fine-grained ZSL, to mitigate the commonly known seen/unseen domain bias and misaligned visual-semantics mapping problems, and have made profound progress. Notably, this paradigm differs from existing close-set fine-grained methods and, therefore, can pose unique and nontrivial challenges. However, to the best of our knowledge, there remains a lack of systematic summaries of this topic. To enrich the literature of this domain and provide a sound basis for its future development, in this work, we present a broad review of recent advances for fine-grained analysis in ZSL. Concretely, we first provide a taxonomy of existing methods and techniques with a thorough analysis of each category. Then, we summarize the benchmark, covering publicly available datasets, models, implementations, and some more details as a library. Last, we sketch out some related applications. In addition, we discuss vital challenges and suggest potential future directions.

Note: this is a collection of representative fine-grained zero-shot learning methods, covering publicly available datasets, models, implementations, etc. For more detailed information, refer to the related Survey Paper.

Please feel free to contact us (jingcai.guo@ieee.org) if you have any advice.

Datasets

• CUB: Caltech-UCSD Birds 200 [Download]
• FLO: Automated Flower Classification over a Large Number of Classes [Download]
• SUN: SUN attribute database: Discovering, annotating, and recognizing scene attributes [Download]
• NABirds: Building a bird recognition app and large scale dataset with citizen scientists: The fine print in fine-grained dataset collection [Download]
• DeepFashion: DeepFashion: Powering Robust Clothes Recognition and Retrieval with Rich Annotations [Download]
• AWA: Attribute-Based Classification for Zero-Shot Visual Object Categorization [Download]
• AWA2: Zero-Shot Learning—A Comprehensive Evaluation of the Good, the Bad and the Ugly [Download]
• APY: Describing objects by their attributes [Download]

---

Attention-Based Methods

Title	Venue	Backbone	FineTune	Resolution	Datasets	Code
Discriminative learning of latent features for zero-shot recognition	CVPR'18	GoogleNet, VGG19	✅	224x224	CUB, AWA	Code
Attribute Attention for Semantic Disambiguation in Zero-Shot Learning	ICCV'19	GoogleNet, ResNet101, VGG19	✅	224x224	CUB, SUN, AWA2	Code
Attentive Region Embedding Network for Zero-shot Learning	CVPR'19	ResNet101	✅	224x224	CUB, SUN, AWA2, APY	Code
Semantic-Guided Multi-Attention Localization for Zero-Shot Learning	NeurIPS'19	VGG19	✅	448x448	CUB, FLO, AWA	Code
Region Graph Embedding Network for Zero-Shot Learning	ECCV'20	ResNet101	✅	224x224	CUB, SUN, AWA2, APY	Code
Fine-Grained Generalized Zero-Shot Learning via Dense Attribute-Based Attention	CVPR'20	ResNet101	❌	224x224	CUB, SUN, DeepFashion, AWA2	Code
Region Semantically Aligned Network for Zero-Shot Learning	CIKM'21	ResNet101	-	448x448	CUB, SUN, AWA2	Code
Goal-Oriented Gaze Estimation for Zero-Shot Learning	CVPR'21	ResNet101	✅	448x448	CUB, SUN, AWA2	Code
I2DFormer: Learning Image to Document Attention for Zero-Shot Image Classification	NeurIPS'22	ViT-B	✅	224x224	CUB, FLO, AWA2	Code
MSDN: Mutually Semantic Distillation Network for Zero-Shot Learning	CVPR'22	ResNet101	❌	448x448	CUB, SUN, AWA2	Code
TransZero: Attribute-Guided Transformer for Zero-Shot Learning	AAAI'22	ResNet101	❌	448x448	CUB, SUN, AWA2	Code
DUET: Cross-Modal Semantic Grounding for Contrastive Zero-Shot Learning	AAAI'23	ViT-B	✅	224x224	CUB, SUN, AWA2	Code
Progressive Semantic-Visual Mutual Adaption for Generalized Zero-Shot Learning	CVPR'23	ViT-B	✅	224x224	CUB, SUN, AWA2	Code

Non-Attention Methods

Prototype Learning

Title	Venue	Backbone	FineTune	Resolution	Datasets	Code
Attribute Prototype Network for Zero-Shot Learning	NeurIPS'20	ResNet101	✅	224x224	CUB, SUN, AWA2	Code
Dual Progressive Prototype Network for Generalized Zero-Shot Learning	NeurIPS'21	ResNet101	✅	448x448	CUB, SUN, AWA2, APY	Code
Dual Part Discovery Network for Zero-Shot Learning	MM'22	ResNet101	❌	448x448	CUB, SUN, AWA2	Code
Boosting Zero-shot Learning via Contrastive Optimization of Attribute Representations	TNNLS'23	ResNet101, ViT-L	✅	224x224, 448x448	CUB, SUN, AWA2	Code

Data Manipulation

Title	Venue	Backbone	FineTune	Resolution	Datasets	Code
Link the head to the “beak”: Zero Shot Learning from Noisy Text Description at Part Precision	CVPR'17	VGG16	❌	-	CUB, NABirds	Code
Stacked Semantics-Guided Attention Model for Fine-Grained Zero-Shot Learning	NeurIPS'18	VGG16	❌	-	CUB, NABirds	Code
Semantic-guided Reinforced Region Embedding for Generalized Zero-Shot Learning	AAAI'21	ResNet101	-	448x448	CUB, SUN, AWA2, APY	Code
VGSE: Visually-Grounded Semantic Embeddings for Zero-Shot Learning	CVPR'22	ResNet50	-	-	CUB, SUN, AWA2	Code

Graph Modeling

Title	Venue	Backbone	FineTune	Resolution	Datasets	Code
Attribute Propagation Network for Graph Zero-Shot Learning	AAAI'20	ResNet101	❌	-	CUB, SUN, AWA, AWA2, APY	Code
GNDAN: Graph Navigated Dual Attention Network for Zero-Shot Learning	TNNLS'22	ResNet101	❌	448x448	CUB, SUN, AWA2	Code
Graph Knows Unknowns: Reformulate Zero-Shot Learning as Sample-Level Graph Recognition	AAAI'23	ResNet34	-	-	CUB, NABirds	Code
Explanatory Object Part Aggregation for Zero-Shot Learning	TPAMI'23	AlexNet, ResNet50	✅	-	CUB, SUN, FLO, AWA2	Code

Generative

Title	Venue	Backbone	FineTune	Resolution	Datasets	Code
A Generative Adversarial Approach for Zero-Shot Learning from Noisy Texts	CVPR'18	VGG16	❌	224x224	CUB, NABirds	Code
Compositional Zero-Shot Learning via Fine-Grained Dense Feature Composition	NeurIPS'20	ResNet101	❌	224x224	CUB, SUN, DeepFashion, AWA2	Code
Zero-Shot Learning With Attentive Region Embedding and Enhanced Semantics	TNNLS'22	ResNet101	❌	224x224	CUB, SUN, AWA, AWA2, APY	Code

Attribute Selection

Title	Venue	Backbone	FineTune	Resolution	Datasets	Code
Multi-Cue Zero-Shot Learning with Strong Supervision	CVPR'16	VGG16	❌	224x224	CUB	Code

Citation

@article{guo2024fine,
  author    = {Jingcai Guo and
               Zhijie Rao and
               Zhi Chen and
               Jingren Zhou and
               Dacheng Tao},
  title     = {Fine-Grained Zero-Shot Learning: Advances, Challenges, and Prospects},
  journal   = {arXiv preprint arXiv:2401.17766},
  year      = {2024},
  url       = {https://arxiv.org/abs/2401.17766}
}