Skip to content
Obj-GAN - Official PyTorch Implementation
Python C Cuda Other
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
box_generation add the code for producing the meta data Jul 23, 2019
data/coco add image and anno folders Jun 3, 2019
image_generation add imports Jul 23, 2019
shape_generation modify pretrained path Jun 3, 2019
README.md Update README.md Jul 23, 2019
example.png Add files via upload May 29, 2019
framework.png Add files via upload May 29, 2019
step_vis.png Add files via upload Jun 10, 2019

README.md

Obj-GAN

Obj-GAN - Official PyTorch Implementation

Pytorch implementation for reproducing Obj-GAN results in the paper Object-driven Text-to-Image Synthesis via Adversarial Training by Wenbo Li*, Pengchuan Zhang*, Lei Zhang, Qiuyuan Huang, Xiaodong He, Siwei Lyu, Jianfeng Gao. (This work was performed when Wenbo was an intern with Microsoft Research).

Picture: If you are asked to draw a picture of several people in their ski gear are in the snow, chances are you will start with an outline of four persons with positioned reasonably in the center of the canvas, then add a sketch of the skis under their feet. Although it is not mentioned in the description, you may decide to add one backpack to each of them to match our common sense. Finally, you carefully finish the details, like painting their clothes blue, their scarves pink and the background white, to make the persons more realistic and the background matches the description better. To make the scene more vivid, you might sketch some brown stones in the snow which indicates that they are in mountains.

Now, there’s a bot that can do that, too.

Dependencies

python 3.6

Pytorch 0.4.1

In addition, please add the project folder to PYTHONPATH and pip install the following packages:

  • python-dateutil
  • easydict
  • pandas
  • torchfile
  • nltk
  • scikit-image
  • spacy
  • PyYAML
  • cffi
  • torchtext
  • dill
  • Cython

Data

  1. Download our preprocessed metadata for coco and merge them to data/coco
  2. Download coco dataset, extract the images to data/coco/images, and extract the annotations to data/coco/insanns

Training

  • Train box generator:
    • cd box_generation
    • python sample.py --is_training 1
  • Train shape generator:
    • cd shape_generation
    • ./make.sh
    • python main.py --gpu '0,1' --FLAG
  • Train image generator:
    • cd image_generation
    • ./make.sh
    • python main.py --gpu '0,1' --FLAG

Pretrained Model

Download and save them to data/coco/pretrained/

Note that we have made some modifications (changing the obj attention estimation from "dot product between Glove embeddings" to "cosine similarity between Glove embeddings") based on the code for CVPR submission, and trained 120 epochs using batch size 16. Compared to the results in the paper, the updated results are better on FID and R-prsn scores, and worse on Inception score (because we do not get a chance to train the model using larger batch size).

Methods Inception ⬆️ FID ⬇️ R-prsn ⬆️
Obj-GAN (pred box & pred shp) 27.32 ± 0.40 24.70 91.91 ± 2.37
Obj-GAN (gt box & pred shp) 28.22 ± 0.35 22.67 93.00 ± 2.15
Obj-GAN (gt box & gt shp) 31.01 ± 0.27 17.03 94.42 ± 2.03

Tips for optimizing the Inception score (though it is boring):

  • Increase the batch size as large as possible via distributed training
  • Increase the weight for the DAMSM loss

Sampling

  • Run box generator:
    • cd box_generation
    • python sample.py --is_training 0 --load_checkpoint [replace with your ckpt path]
  • Run shape generator:
    • cd shape_generation
    • python main.py --gpu '0,1' --NET_G [replace with your ckpt path]
  • Run image generator:
    • cd image_generation
    • python main.py --gpu '0,1' --NET_G [replace with your ckpt path]

More Results

Citing Obj-GAN

If you find Obj-GAN useful in your research, please consider citing:

@article{objgan19,
  author    = {Wenbo Li, Pengchuan Zhang, Lei Zhang, Qiuyuan Huang, Xiaodong He, Siwei Lyu, Jianfeng Gao},
  title     = {Object-driven Text-to-Image Synthesis via Adversarial Training},
  Year = {2019},
  booktitle = {{CVPR}}
}
You can’t perform that action at this time.