VL-GPT: A Generative Pre-trained Transformer for Vision and Language Understanding and Generation
Sijie Zhao2, Hengshuang Zhao3, Xiaohua Wang1, Ying Shan2
* Equal Contribution
-
VL-GPT is a generative pre-trained transformer model for vision and language understanding and generation tasks, which can perceive and generate visual and linguistic data concurrently. By employing a straightforward auto-regressive objective, VL-GPT achieves a unified pre-training for both image and text modalities.
-
We also propose an image tokenizer-detokenizer framework for the conversion between raw images and continuous visual embeddings, analogous to the role of the BPE tokenization in language models.
- Training and evaluation code
- Pretrained and instruction-tuned model weights
This project is released under the Apache 2.0 license. Please see the LICENSE file for more information.