05/28/2020: Released finetuned models on downstream tasks, please check MODEL_ZOO.md. 05/15/2020: Released pretrained models, datasets, and code for downstream tasks finetuning.
This repository contains source code necessary to reproduce the results presented in the paper Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks. We propose a new cross-modal pre-training method Oscar (Object-Semantics Aligned Pre-training). It leverages object tags detected in images as anchor points to significantly ease the learning of image-text alignments. We pre-train Oscar on the public corpus of 6.5 million text-image pairs, and fine-tune it on downstream tasks, creating new state-of-the-arts on six well-established vision-language understanding and generation tasks. For more on this project, see the Microsoft Research Blog post.
| Task | t2i | t2i | i2t | i2t | IC | IC | IC | IC | NoCaps | NoCaps | VQA | NLVR2 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Metric | R@1 | R@5 | R@1 | R@5 | B@4 | M | C | S | C | S | test-std | test-P |
| SoTA_S | 39.2 | 68.0 | 56.6 | 84.5 | 38.9 | 29.2 | 129.8 | 22.4 | 61.5 | 9.2 | 70.90 | 53.50 |
| SoTA_B | 48.4 | 76.7 | 63.3 | 87.0 | 39.5 | 29.3 | 129.3 | 23.2 | 73.1 | 11.2 | 72.54 | 78.87 |
| SoTA_L | 51.7 | 78.4 | 66.6 | 89.4 | - | - | - | - | - | - | 73.40 | 79.50 |
| ----- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Oscar_B | 54.0 | 80.8 | 70.0 | 91.1 | 40.5 | 29.7 | 137.6 | 22.8 | 78.8 | 11.7 | 73.44 | 78.36 |
| Oscar_L | 57.5 | 82.8 | 73.5 | 92.2 | 41.7 | 30.6 | 140.0 | 24.5 | 80.9 | 11.3 | 73.82 | 80.05 |
| gain | 5.8 | 4.4 | 6.9 | 2.8 | 2.2 | 1.3 | 10.7 | 1.3 | 7.8 | 0.5 | 0.42 | 0.55 |
t2i: text-to-image retrieval; i2t: image-to-text retrieval; IC: image captioning on COCO.
We released pre-trained models and datasets for downstream tasks. Please check DOWNLOAD.md for details.
Check INSTALL.md for installation instructions.
Check MODEL_ZOO.md for scripts to run each downstream tasks and the expected performance.
Please consider citing this paper if you use the code:
@article{li2020oscar,
title={Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks},
author={Li, Xiujun and Yin, Xi and Li, Chunyuan and Hu, Xiaowei and Zhang, Pengchuan and Zhang, Lei and Wang, Lijuan and Hu, Houdong and Dong, Li and Wei, Furu and Choi, Yejin and Gao, Jianfeng},
journal={arXiv preprint arXiv:2004.06165},
year={2020}
}
Oscar is released under the MIT license. See LICENSE for details.