Skip to content

ixuan789/RAT-GAN-main

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

2 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Recurrent-Affine-Transformation-for-Text-to-image-Synthesis

Official Pytorch implementation for our paper Recurrent-Affine-Transformation-for-Text-to-image-Synthesis

image

Examples

图片


Requirements

  • python 3.8
  • Pytorch 1.11.0+cu113
  • easydict
  • nltk
  • scikit-image
  • A 2080 TI (set nf=32 in *.yaml) or a 3090 32GB (set nf=64 in *.yaml)

Note that nf=32 produces a IS around 5.0 on CUB. To reproduce the final results, please use a GPU more than 32GB.

Installation

Clone this repo.

git clone https://github.com/senmaoy/Recurrent-Affine-Transformation-for-Text-to-image-Synthesis.git
cd Recurrent-Affine-Transformation-for-Text-to-image-Synthesis/code/

Datasets Preparation

  1. Download the preprocessed metadata for birds coco and save them to data/
  2. Download the birds image data. Extract them to data/birds/.Raw text data of CUB dataset is avaiable here
  3. Download coco dataset and extract the images to data/coco/
  4. Download flower dataset and extract the images to data/flower/. Raw text data of flower dataset is avaiable here

Note that flower dataset is a bit different from cub and coco with a standalone dataset processing script.

It's easy to train on your own Datasets (similar to the processing for flower dataset)

  1. Prepare a captions.pickle containing all the image paths. Note that captions.pickle should be prepared by yourself.
  2. Save captions.pickle under data_dir.
  3. Put all the captions of an image in a standalone txt file (one caption per line). This txt file will be later read by dataset_flower.py in line 149: cap_path = '%s/%s.txt' % ('/home/yesenmao/dataset/flower/jpg_text/', filenames['img'][i])
  4. Run main.py as usual. Dataset_flower.py will automatically process your own dataset.

Pre-trained text encoder

  1. Download the pre-trained text encoder for CUB and save it to ../bird/
  2. Download the pre-trained text encoder for coco and save it to ../bird/
  3. Download the pre-trained text encoder for flower and save it to ../bird/

Training

Train RAT-GAN models:

  • For bird dataset: python main.py --cfg cfg/bird.yml

  • For coco dataset: python main.py --cfg cfg/coco.yml

  • For flower dataset: python main.py --cfg cfg/flower.yml

  • *.yml files are example configuration files for training/evaluation our models.

Evaluating

Dwonload Pretrained Model

Evaluate RAT-GAN models:


Citing RAT-GAN

If you find RAT-GAN useful in your research, please consider citing our paper:

@article{ye2022recurrent,
  title={Recurrent Affine Transformation for Text-to-image Synthesis},
  author={Ye, Senmao and Liu, Fei and Tan, Minkui},
  journal={arXiv preprint arXiv:2204.10482},
  year={2022}
}

If you are interseted, join us on Wechat group where a dozen of t2i partners are waiting for you!

image

The code is released for academic research use only. Please contact me through senmaoy@gmail.com

Reference

About

No description, website, or topics provided.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages