Skip to content
/ SADE Public

An Examination of the Compositionality of Large Generative Vision-Language Models

License

Notifications You must be signed in to change notification settings

TeleeMa/SADE

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

4 Commits
 
 
 
 
 
 
 
 

Repository files navigation

An Examination of the Compositionality of Large Generative Vision-Language Models

Teli Ma1, Rong Li1, Junwei Liang1, 2

1 AI Thrust, HKUST(GZ), 2 CSE, HKUST

[paper] [homepage]

Abstract

With the success of Large Language Models (LLMs), many Generative Vision-Language Models (GVLMs) have been constructed via multimodal instruction tuning. However, the performance of GVLMs in multimodal compositional reasoning remains under-explored. In this paper, we examine both the evaluation metrics ( VisualGPTScore, etc.) and current benchmarks for evaluating the compositionality of GVLMs. We identify the syntactical bias in current benchmarks, which is exploited by the linguistic capability of GVLMs. The bias renders VisualGPTScore an insufficient metric for assessing GVLMs. To combat this, we first introduce a SyntaxBias Score, leveraging LLMs to quantify such bias for mitigation. A challenging new task is subsequently added to evaluate the robustness of GVLMs against inherent inclination toward syntactical correctness. Using the bias-mitigated datasets and the new task, we propose a novel benchmark, namely SyntActically DE-biased benchmark (SADE). Our study provides an unbiased benchmark for the compositionality of GVLMs, facilitating future research in this direction.

Dataset Download

  1. Download the VG, VL-CheckList, COCO2014, Flickr30k, Winoground.

  2. Download the positive and negative references of SADE from here.

License

MODE is released under the MIT License.

Citation

@article{ma2023examination,
  title={An Examination of the Compositionality of Large Generative Vision-Language Models},
  author={Ma, Teli and Li, Rong and Liang, Junwei},
  journal={arXiv preprint arXiv:2308.10509},
  year={2023}
}

About

An Examination of the Compositionality of Large Generative Vision-Language Models

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published