Towards Zero-Shot Diabetic Retinopathy Grading: Learning Generalized Knowledge via Prompt-Driven Matching and Emulating
This is an official implementation of the following paper:
Huan Wang, Haoran Li, Yuxin Lin, Huaming Chen, Jun Yan, Lijuan Wang, Jiahua Shi, Qihao Xu, Yongting Hu, Yong Xu*, Jun Shen*. "Towards Zero-Shot Diabetic Retinopathy Grading: Learning Generalized Knowledge via Prompt-Driven Matching and Emulating". The Association for the Advancement of Artificial Intelligence (AAAI), AAAI 2026 (Oral).
Abstract: As one of the primary causes of visual impairment, Diabetic Retinopathy (DR) requires accurate and robust grading to facilitate timely diagnosis and intervention. Different from traditional DR grading methods that utilize single-view images, recent clinical studies have revealed that multi-view fundus images can significantly enhance DR detection performance by expanding the field of view (FOV). However, the inherent class imbalance in fundus image datasets often results in a long-tailed distribution, leading to an overwhelming presence of head categories. To address this challenge, we adopt a zero-shot learning paradigm, aiming to mitigate the long-tailed problem by enhancing the model's generalization capability to unseen classes. In this paper, we explore the potential of using prompt tuning for zero-shot DR grading with multi-view fundus images and introduce a new ProME-DR framework, based on Prompt Matching and Emulating, to handle the unseen DR categories and views beyond the training set. ProME-DR decouples the learning process into two stages, enabling generalization to novel categories. Initially, it leverages two sets of distinct prompt units to capture semantic and inter-view consistency knowledge via a split-and-mask way, gathering instance-level vision clues. Subsequently, it refines another set of context prompt units generated by a concept-aware emulator, linking extensible knowledge learned from the previously seen attributes for zero-shot DR grading. Extensive experiments conducted on 8 datasets (e.g., MFIDDR, the latest multi-view fundus image dataset) and various zero-shot scenarios demonstrate the effectiveness of the proposed ProME-DR, surpassing popular counterparts.
Here is an example to run ProME-DR:
python3 main_ProMEDR.py -stage1_epochs 10 -generator_layer 2 \
-debug_mode 0 -es 10 -e 100 -data MFIDDR -bs 64 -t_bs 100 -opt adam -lr 0.0001 -mask_prob 0.7 \
-use_NTP 0 -tp_N_CTX 16 -CLS_NUM_TOKENS 5 -VIEW_NUM_TOKENS 4 -GP_CLS_NUM_TOKENS 5 -GP_VIEW_NUM_TOKENS 4