VAR-CLIP:
Text-to-Image Generator with Visual Auto-Regressive Modeling

 

VAR-CLIP: Text-to-Image Generator with Visual Auto-Regressive Modeling
Qian Zhang, Xiangzi Dai, Ninghua Yang, Xiang An, Ziyong Feng, Xingyu Ren
Institute of Applied Physics and Computational Mathematics, DeepGlint,Shanghai Jiao Tong University

Some example for text-conditional generation:

.

Some example for class-conditional generation:

.

TODO

• Relased train code.
• Relased Arxiv.
• Training T2I on the ImageNet dataset has been completed.
• Training on the ImageNet dataset has been completed.

Getting Started

Requirements

pip install -r requirements.txt

Download Pretrain model/Dataset

1. Place the downloaded ImageNet train/val parts separately under train/val in the directory ./imagenet/
2. Download clip/vae pretrain model put on pretrained/

Download ClIP_L14
Download VAE

Training Scripts

#training VAR-CLIP-d16 for 1000 epochs on ImageNet 256x256 costs 4.1 days on 64 A100s
#Before running, you need to configure the IP addresses of multiple machines in the run.py file and data_path
python run.py

demo Scripts

#you can run demo_samle.ipynb get text-conditional generation resulets after train completed.
demo_sample.ipynb

License

This project is licensed under the MIT License - see the LICENSE file for details.

Citations

@misc{zhang2024varcliptexttoimagegeneratorvisual,
      title={VAR-CLIP: Text-to-Image Generator with Visual Auto-Regressive Modeling}, 
      author={Qian Zhang and Xiangzi Dai and Ninghua Yang and Xiang An and Ziyong Feng and Xingyu Ren},
      year={2024},
      eprint={2408.01181},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2408.01181}, 
}

• VAR - https://github.com/FoundationVision/VAR
• CLIP - https://github.com/openai/CLIP