[WACV'18] Where and Who? Automatic Semantic-Aware Person Composition
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Type Name Latest commit message Commit time
Failed to load latest commit information.


Where and Who? Automatic Semantic-Aware Person Composition

Fuwen Tan, Crispin Bernier, Benjamin Cohen, Vicente Ordonez, Connelly Barnes, WACV 2018.


Image compositing is a method used to generate realistic yet fake imagery by inserting contents from one image to another. Previous work in compositing has focused on improving appearance compatibility of a user selected foreground segment and a background image (i.e. color and illumination consistency). In this work, we instead develop a fully automated compositing model that additionally learns to select and transform compatible foreground segments from a large collection given only an input image background. To simplify the task, we restrict our problem by focusing on human instance composition, because human segments exhibit strong correlations with their background and because of the availability of large annotated data. We develop a novel branching Convolutional Neural Network (CNN) that jointly predicts candidate person locations given a background image. We then use pre-trained deep feature representations to retrieve person instances from a large segment database. Experimental results show that our model can generate composite images that look visually convincing.

Software required

  • Python 2.7
  • Tensorflow (1.4.1 or above)
  • Keras (2.0.8 or above)

Installation for the demo

  1. Clone the repository
git clone https://github.com/fwtan/who_where.git

We'll call the directory that you cloned the repo into COMP_ROOT

  1. Compile the Cython and pycocotools modules

    cd $COMP_ROOT/lib
  2. Download the auxiliary data, pretrained model and example inputs

    cd $COMP_ROOT/tools

    This will populate the $COMP_ROOT/data folder with coco, pretrained, and testset.

  3. Download the COCO 2014 validation set and the annotations if you have not done so

    cd $COMP_ROOT/tools

    This will populate the $COMP_ROOT/data folder with coco/images and coco/annotations. The COCO validation data is used to help build candidate pool for segment retrieval.


After the installation, you should be able to run the demo. To run the demo

cd $COMP_ROOT/tools
python demo.py

You can find the output composite images in $COMP_ROOT/output/composite_colors.

Example outputs

Here the first row shows the input images; the second row shows the heatmaps of the bounding box prediction, in which the green boxes indicate the top-1 predictions; the third row shows the composite outputs.

Test with your own images

As the model takes both the color and layout image as input, if you'd like to test with your own images, you may have to collect the object detection results on the input images first.

We provide an example script $COMP_ROOT/tools/collect_detections.py to collect the detection outputs from the Faster RCNN system (https://github.com/rbgirshick/py-faster-rcnn). Example outputs are also included in the directory $COMP_ROOT/data/testset/test_detections.

For each image, the detection output is a JSON file containing the bounding boxes and categories of the detected objects. The class IDs we use are the same as the Faster RCNN system.

Once the detection outputs are available, the $COMP_ROOT/tools/create_layouts.py script could help render the detections as layout images.

Please contact Fuwen Tan (fuwen.tan@virginia.edu) if you have any questions.


If you find our paper/code useful, please consider citing:

    title={Where and Who? Automatic Semantic-Aware Person Composition},
    author={Tan, Fuwen and Bernier, Crispin and Cohen, Benjamin and Ordonez, Vicente and Barnes, Connelly},
    booktitle={IEEE Winter Conf. on Applications of Computer Vision (WACV)},