Skip to content

MM2022 Workshop-Perceptual Conversational Head Generation with Regularized Driver and Enhanced Renderer

License

Notifications You must be signed in to change notification settings

hzwer/MM2022-ViCoPerceptualHeadGeneration

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

11 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Perceptual Conversational Head Generation with Regularized Driver and Enhanced Renderer

Zhewei Huang, Ailin Huang, Shuchang Zhou

image

Introduction

This project is the implement of Perceptual Conversational Head Generation with Regularized Driver and Enhanced Renderer. We ranked first place in the listening head generation track and second place in the talking head generation track in the official ranking of MM2022-ViCo Conversational Head Generation Challenge. Our team name is Megvii_goodjuice.

The whole pipeline of challenge can be found on vico challenge baseline. We currently provide our major modification of the baseline.

Contributors: @P2Oileen, @hzwer

Modification

  • Architecture & Pipeline

Clone vico challenge baseline, replace vico_challenge_baseline/vico/networks/audio_to_face.py, vico_challenge_baseline/vico/networks/speaker_generator.py, vico_challenge_baseline/PIRender/inference.py, and move u2net.py to vico_challenge_baseline/PIRender/u2net.py.

  • Fusion

Use U2Net to segment the backgrounds and fuse them to remain the background unchanged. You should also download U2Net weights u2net_human_seg.pth from Google Drive and save in vico_challenge_baseline/PIRender/u2net_human_seg.pth.

  • Image Boundary Inpainting

Change the padding mode of grid_sample in PIRenderer from "zeros" to "border".

Citation

@InProceedings{huang2022perceptual,
  title={Perceptual Conversational Head Generation with Regularized Driver and Enhanced Renderer},
  author={Huang, Ailin and Huang, Zhewei and Zhou, Shuchang},
  booktitle={Proceedings of the 30th ACM International Conference on Multimedia (MM'22)},
  year={2022}
}

The ViCo baseline method: 
@InProceedings{zhou2022responsive,
    title={Responsive Listening Head Generation: A Benchmark Dataset and Baseline},
    author={Zhou, Mohan and Bai, Yalong and Zhang, Wei and Yao, Ting and Zhao, Tiejun and Mei, Tao},
    booktitle={Proceedings of the European conference on computer vision (ECCV)},
    year={2022}
}

About

MM2022 Workshop-Perceptual Conversational Head Generation with Regularized Driver and Enhanced Renderer

Topics

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages