CharacterLipSync

Real-Time Lip Sync for Live 2D Animation

Paper: https://arxiv.org/abs/1910.08685

Video summary: character_lipsync_video_summary.mp4

Additional results:

The spoken sentences are taken from a test set of 50 recordings, which we used to generate side-by-side comparisons that we ran on Amazon Mechanical Turk.

Ch - Adobe Character Animator v1. https://www.adobe.com/products/character-animator.html

ToonBoom https://www.toonboom.com/products/harmony

Our result is always on the RIGHT side. Following video comparisons are included:

Baseline 1 : Ch Online (live) vs Ours (live)

1. ChOnline_vs_Ours_Female
2. ChOnline_vs_Ours_Kid
3. ChOnline_vs_Ours_Male

Baseline 2 : Ch Offline (not live) vs Ours (live)

1. ChOffline_vs_Ours_Cartoon
2. ChOffline_vs_Ours_Female
3. ChOffline_vs_Ours_Kid
4. ChOffline_vs_Ours_Male

Baseline 3 : ToonBoom Offline (not live) vs Ours (live)

1. ToonBoom_vs_Ours_Cartoon
2. ToonBoom_vs_Ours_Female
3. ToonBoom_vs_Ours_Male

Baseline 4 : OursNoAug vs Ours

1. OursNoAug_vs_Ours_Female
2. OursNoAug_vs_Ours_Kid
3. OursNoAug_vs_Ours_Male

Baseline 5 : Ours4 vs Ours

1. Ours4_vs_Ours_Female
2. Ours4_vs_Ours_Male