We proposed a method to detect a saliency map of the object and manipulate the background of the given object. Saliency detection is identifying the most visually distinctive objects in an image. It is useful as a pre-processing step for many computer vision tasks. Recently, great features from convolutional neural network (CNN) has shown considerable progress in this area. However, CNN-based saliency detection results blurry map near the boundary of salient objects. Applying conditional random field (CRF) in super-pixel domain can refine the saliency map from CNN, and make sharp and fine boundaries. Generative background manipulation is the scheme that neural networks find a different and suitable background for the given object. To manipulate the background for the object, we learn the probability model of the dataset by generative adversarial networks. In our project, we first present the saliency detection using CNN and CRF, then we manipulate the saliency map by transferring saliency map to the image domain using Generative Adversarial Network(GAN).
First, using CNN and CRF the saliency map is detected. Then the GAN, which is trained on the same dataset, generates background and foreground for the input image.
Figure 1. Overall scheme of proposed method
Dataset: HKU-IS dataset
Results of saliency map detection using CNN and CRF
Figure 2. Visual comparison on the HKU-IS dataset: (from left top to right bottom) input, YS, MDF, DCL and ground truth images.
Results of background generation using GAN
Figure 3. Background Manipulation Result on the HKU-IS dataset: (from left to right) saliency map, generated background, and ground truth.
For more details, please check out the paper.