Skip to content
Real-Time Lip Sync for Live 2D Animation
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.

README.md

CharacterLipSync

Real-Time Lip Sync for Live 2D Animation

Alt text

Paper: https://arxiv.org/abs/1910.08685

Video summary: character_lipsync_video_summary.mp4

Additional results:

The spoken sentences are taken from a test set of 50 recordings, which we used to generate side-by-side comparisons that we ran on Amazon Mechanical Turk.

Ch - Adobe Character Animator v1. https://www.adobe.com/products/character-animator.html

ToonBoom https://www.toonboom.com/products/harmony

Our result is always on the RIGHT side. Following video comparisons are included:

Baseline 1 : Ch Online (live) vs Ours (live)

  1. ChOnline_vs_Ours_Female
  2. ChOnline_vs_Ours_Kid
  3. ChOnline_vs_Ours_Male

Baseline 2 : Ch Offline (not live) vs Ours (live)

  1. ChOffline_vs_Ours_Cartoon
  2. ChOffline_vs_Ours_Female
  3. ChOffline_vs_Ours_Kid
  4. ChOffline_vs_Ours_Male

Baseline 3 : ToonBoom Offline (not live) vs Ours (live)

  1. ToonBoom_vs_Ours_Cartoon
  2. ToonBoom_vs_Ours_Female
  3. ToonBoom_vs_Ours_Male

Baseline 4 : OursNoAug vs Ours

  1. OursNoAug_vs_Ours_Female
  2. OursNoAug_vs_Ours_Kid
  3. OursNoAug_vs_Ours_Male

Baseline 5 : Ours4 vs Ours

  1. Ours4_vs_Ours_Female
  2. Ours4_vs_Ours_Male
You can’t perform that action at this time.