Awesome Zero-shot Captioning

A curated list of zero-shot captioning papers, including image-to-text generation, video-to-text generation. Maintained by Junjie Fei (junjiefei@outlook.com). Most recently updated on 2023/08/26.

Image-to-Text Generation

2023

• Transferable Decoding with Visual Entities for Zero-Shot Image Captioning

ICCV 2023 [paper] [code]

• ConZIC: Controllable Zero-Shot Image Captioning by Sampling-Based Polishing

CVPR 2023 [paper] [code] [demo]

• DeCap: Decoding CLIP Latents for Zero-Shot Captioning via Text-Only Training

ICLR 2023 [paper] [code]

• MultiCapCLIP: Auto-Encoding Prompts for Zero-Shot Multilingual Visual Captioning

ACL 2023 [paper] [code]

• ZeroGen: Zero-shot Multimodal Controllable Text Generation with Multiple Oracles

arXiv 2023 [paper] [code]

• Re-ViLM: Retrieval-Augmented Visual Language Model for Zero and Few-Shot Image Captioning

arXiv 2023 [paper]

• Test-Time Adaptation with CLIP Reward for Zero-Shot Generalization in Vision-Language Models

arXiv 2023 [paper] [project page]

• CoBIT: A Contrastive Bi-directional Image-Text Generation Model

arXiv 2023 [paper]

2022

• ZeroCap: Zero-Shot Image-to-Text Generation for Visual-Semantic Arithmetic

CVPR 2022 [paper] [code] [demo]

• Text-Only Training for Image Captioning using Noise-Injected CLIP

EMNLP 2022 [paper] [code]

• Visual Information Guided Zero-Shot Paraphrase Generation

COLING 2022 [paper]

• Language Models Can See: Plugging Visual Controls in Text Generation

arXiv 2022 [paper] [code] [demo]

• Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language

arXiv 2022 [paper] [code] [project page]

• Large-Scale Bidirectional Training for Zero-Shot Image Captioning

arXiv 2022 [paper]

Before 2022

• Image Captioning with Unseen Objects

BMVC 2019 [paper]

Video-to-Text Generation

2023

• Zero-Shot Dense Video Captioning by Jointly Optimizing Text and Momen

arXiv 2023 [paper]

2022

• Zero-Shot Video Captioning with Evolving Pseudo-Tokens

arXiv 2022 [paper] [code]

Before 2022

• Learning to Compose Topic-Aware Mixture of Experts for Zero-Shot Video Captioning

AAAI 2019 [paper]