This repository contains the official code for the ICCV 2023 paper:
I can't believe there's no Images! Learning Visual Tasks Using only Language Supervision
This project trains models on pure-text data and then shows they can applied to the same tasks with visual inputs instead of text, thus demonstrating zero-shot cross-modal transfer. This is done by using the shared semantic embedding space of contrastive vision and language models.
More details can be found on our webpage.
Install pytorch, we have tested this code with torch 1.10.1 and 1.11.0 Then install the other requirements with:
pip install -r requirements.txt
Finally download the needed datasets, data should be saved in the paths stored in close/file_paths.py. The data can be downloaded automatically using this script:
python close/download.py
This will download about 45G of data. Data can also be manually downloaded these sources:
The 2014 train and validation images for COCO should be put in ~/data/coco/images The annotations should be saved to ~/data/coco/annotations.
We use the Karpathy Split found here The coco_dataset.json file should be put into ~/data/coco.
The data build from here. By default the annotations should be in ~/data/SNLI_VE and the images should be in ~/data/SNLI_VE/Flickr30K
VQA annotation from here, which should be saved ~/data/vqa2
Download the files from here, by default the files should be put into ~/data/vqa-e
We use the Visual News dataset, a large corpus consisting of both news images and articles from several news sources. More details can be found here.
Email sophiag[at]allenai[dot]org for trained models on visual news.
The linear and covariance adapters we used in our ablations can be downloaded directly from
AWS, see download_adapters in coco/close/download.py
Training is done with cross/experiments/train.py,
python close/experiments/train.py --data {vqa|vqa-e|ve|s-cap|m-cap} --output_dir path/to/output/dir
The data controls which dataset to train on, s-cap is captioning in the single captioning
setting and m-cap is the multiple captioning setting.
The script will use our default values, see the command line args for how to change the parameters.
The evaluations can be done with eval.py, for example:
python coco/experiments/eval.py path/to/model evqa --output_name default
Each model includes a model.json and state-ep8.pth file, the E-VQA model can be downloaded like this:
mkdir model
mkdir model/r0
wget https://ai2-prior-close.s3.us-west-2.amazonaws.com/models/evqa/model.json -O model/model.json
wget https://ai2-prior-close.s3.us-west-2.amazonaws.com/models/evqa/r0/state-ep8.pth -O model/r0/state-ep8.pth
Other models can be downloaded by replace evqa with these names:
- s-cap: Captioning (Single):
- m-cap: Captioning (Multiple):
- vqa: VQA (note trained on train+val):
- evqa: EVQA:
- ve: Visual Entailment: