CTrGAN: Cycle Transformers GAN for Gait Transfer

This repository contains training code for the examples in the WACV 2023 paper "CTrGAN: Cycle Transformers GAN for Gait Transfer ."

CTrGAN transfers the poses of unseen source to the target, while maintaining the natural gait of the target. From left to right: (a) The source’s image is converted to (b) DensePose’s IUV format. (c) The source pose is rendered (using Vid2Vid) to a corresponding RGB image of the target. (c) Our model translates the IUV of the source to the corresponding most natural IUV pose of the target by synthesizing a novel pose. (e) The generated pose is rendered (using Vid2Vid) to a corresponding RGB image of the target.

Note : The code in this repository includes only the CTrGAN code that we used to convert the IUV Pose. It does not include the part that we used to create the Source Pose from the object appearance (Detectron2-Densepose), and the part that we used to convert back from Pose to Appearance (Vid2Vid))

Prerequisites

• Linux
• Python 3.8+
• NVIDIA GPU + CUDA CuDNN

The files requirements.txt environment.yml show the packages we use in the runtime environment.

Data Processing

The code expects to receive iuv images according to the structure shown in ./example/dataset/iuv . Each image will be a png cropped to 256x256 pixels. The object should sit in the center of the image. An image consists of 4 iuv layers and a mask.

Training

Example

python train.py  --name EXP5010 --datafile ./example/configs/train_example.yaml --checkpoints_dir ./example/ --model CTrGAN --ngf 16 --dataset_mode unaligned_sequence --no_dropout --no_flip --loadSize 272 --fineSize 256 --iuv_mode iuv1 --input_nc 4 --output_nc 4 --use_perceptual_loss --pool_size 0 --niter 100 --niter_decay 300 --save_epoch_freq 100 --continue_train --epoch_count 0 --use_sa --use_qsa --seq_len 3 --nThreads 0 --gpu 0

Inference

Example

Note: Be sure to update the variable which_epoch to the value with which you wish to perform the predict. (If the code does not find the model, it will use the initialized parameters of the model, i.e. random)

python predict.py --name EXP5010 --datafile ./example/configs/valid_example.yaml --checkpoints_dir ./example/ --model CTrGAN --ngf 16 --dataset_mode unaligned_sequence --no_dropout --no_flip --loadSize 256 --fineSize 256 --iuv_mode iuv1 --input_nc 4 --output_nc 4 --which_epoch 80 --results_dir ./example/results/ --use_fullseq --seq_len 3 --use_sa --use_qsa --gpu 0

Acknowledgments and Thanks

• This code is based on pytorch-CycleGAN-and-pix2pix and Recycle-GAN :Unsupervised Video Retargeting.
• Portions of the research in this paper uses the CASIA Gait Database collected by Institute of Automation, Chinese Academy of Sciences. link

Cite

If you find it useful, please cite

@InProceedings{Mahpod_2023_WACV,
    author    = {Mahpod, Shahar and Gaash, Noam and Hoffman, Hay and Ben-Artzi, Gil},
    title     = {CTrGAN: Cycle Transformers GAN for Gait Transfer},
    booktitle = {Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)},
    month     = {January},
    year      = {2023},
    pages     = {371-381}
}