DeepFashionSwap

Virtual Faishon TryON - Fashion(Garment) transfer on a given target from a refrence using Generative Adversarial Networks.

Virtual faishon try on is a challenging research area with one of the aim to make shopping more easier. With its development a person can be able to virtually try the desired cloths or faishon items sitting at home. How to acheive it with minimal artifacts, is still a question? Upon several research on different approach I found that the predominant approach to the problem is - To use GAN on combination of key pose of the target person wearing fashion garment along with its segmented image and the Refrence Fashion Item. With this Idea, below I have shown the proposed network architecture for this task.

Proposed Architecture

In the proposed arcitecture, two Encoders and one decoder is used for the Generator Network. Where one encoder is used for Learning features for the corresponding Fused cloth key points and segmented body parts, Whereas other encoder is used to learn the features from the Reference garment. The below image shows the complete proposed architecture, designed by modifying Chao-Te Chou et al. PIVTONS architecture for virtual shoe tryon.

Other Ideas/Networks to use as a Generator are  - 
1. Use of StyleGAN2 architecture, where corresponding keypoints and segmented masked part will be sent as input and Feature of Reference cloth will be given  as a Style vector. 
2. UNet architeccture can be used as a Generator Network.  

Some Methods to generate pose keypoints

1. Openpose pretrained network can be used to generate COCO keypoints for the cloth part. Like shown in below image.

2. Use instance segmentation models for human body parsing like Self-Correction for Human Parsing(SCHP) and Part Grouping Network. Once we have the segmented image, use edge detection technique to identify the edges. Find contours correspoding to cloth part from the edge output.

Related works

Some of the best related research work done in this area with there publication is mentioned below. Do checkout these great research for more insights into Virtual faishon TryON.

1. GarmentGAN: Photo-realistic Adversarial Fashion Transfer
2. ACGPN: Towards Photo-Realistic Virtual Try-On by Adaptively Generating↔Preserving Image Content
3. Styleposegan: Style and Pose Control for Image Synthesis of Humans from a Single Monocular View
4. VOGUE: Try-On by StyleGAN Interpolation Optimization
5. VITON: An Image-based Virtual Try-on Network

Limitations

References

1. Deep Learning for Virtual Try On Clothes – Challenges and Opportunities