Mouth Position Delay / Lag when motion is faster

[AlonDan]

My first few tries where on some random videos and I noticed that the Mouth not always placed on the correct spot.
So I did few experiments trying to replicate the issue before I post it here.

I've downloaded a Head made in Gan2 (random) and made a simple motion with different speeds as a video, so it's supposed to be easier to track compare to a video with angles and what not since it's just a still image moving around in different speeds so you can SEE the visual issue.

ISSUE Description:

1. It seems like when there is a fast motion the mouth will be "behind" the frame like a delay.
2. I don't think it's a track detection issue because it looks like a frame ahead the "position" of the mouth is almost there.

The above are only guess, I only have some experience with other Deep Fake such as:
FaceSwap and DFL so I know that even blurry fast motion could be detected, I just don't know how it works here, probably in a different way but I hope that there is a solution to make even FAST motions trackable to sit on the right position without a lag / delay.

I hope to see this fixed if possible it will be great, please keep up the good work! :)

To keep things clear and simple, the example video is 640x360 - 24fps

Download Video Example Attached as .ZIP file:

result_video.zip

Few Screenshots from the video, shows the mouth not accurate on where it should be:

[prajwalkr]

Hello, thank you for the clear and detailed description of the issue. I think I clearly understand the issue. It is caused by this line:

Wav2Lip/inference.py

Line 97 in 0600d0f

boxes = get_smoothened_boxes(np.array(results), T=5)

This line smooths bounding boxes over a 5 frame window. This is desirable as face detection can be at times be noisy and change a lot across frames. But, when there is rapid motion, the face position changes so much that the averaging the box location across the frames is undesirable.

We will look into this issue and update you once there is some sort of fix.

[prajwalkr]

Can you pull the latest commit, and try with the --nosmooth argument and let me know?

[AlonDan]

Just to be clear: I'm a newbie when it comes to GitHub and I'm not a programmer so everything is very new to me, all I can do is to follow instructions and tutorials.
So I hope that I did it correct:

I've downloaded "inference.py" made a backup of the old one, and put this on the main master directory.

Then I used the same files I did for the test of this issue and at the end I've added --nosmooth like this:
python inference.py --checkpoint_path checkpoints/wav2lip_gan.pth --face MyDir/video.avi --audio MyDir/audio.wav --outfile MyDir/result_video.mp4 --nosmooth

but I get this error:
inference.py: error: unrecognized arguments: --nosmooth

I double checked that I put the NEWEST file (from about 30 minutes ago) instead of the old one, but still it seems like it's not recognizing the new command.

Maybe I did something wrong, not sure if I'm the right man to help testing but I can try if it's something simple enough for me to follow, sorry for the mess.

[dipam7]

Hey, I had a similar issue. I ran it with the --nosmooth argument and it ran till a 100% but then gave me the following error:

Traceback (most recent call last):
  File "inference.py", line 280, in <module>
    main()
  File "inference.py", line 250, in main
    total=int(np.ceil(float(len(mel_chunks))/batch_size)))):
  File "/usr/local/lib/python3.6/dist-packages/tqdm/std.py", line 1127, in __iter__
    for obj in iterable:
  File "inference.py", line 111, in datagen
    face_det_results = face_detect(frames) # BGR2RGB for CNN face detection
  File "inference.py", line 101, in face_detect
    results = [[image[y1: y2, x1:x2], (y1, y2, x1, x2)] for image, (x1, y1, x2, y2) in zip(images, boxes)]
UnboundLocalError: local variable 'boxes' referenced before assignment

I see that line 101 uses boxes

Wav2Lip/inference.py

Line 101 in 8549b79

results = [[image[y1: y2, x1:x2], (y1, y2, x1, x2)] for image, (x1, y1, x2, y2) in zip(images, boxes)]

which we only retrieve if line 100 is true otherwise we don't set it to anything.

Wav2Lip/inference.py

Line 100 in 8549b79

if not args.nosmooth: boxes = get_smoothened_boxes(np.array(results), T=5)

[prajwalkr]

@AlonDan can you check now if it is better?

[AlonDan]

Sure, I just tried it now it won't start running there is an error right after I press ENTER:

File "inference.py", line 6
    <!DOCTYPE html>
    ^
SyntaxError: invalid syntax

Even if I just run the main command:
python inference.py

I get the same error.

[prajwalkr]

Hello, I think this is an error from your end, as you can see that the repo does not have that in line 6:

Wav2Lip/inference.py

Line 6 in 11a8eac

from glob import glob

[AlonDan]

Hmm, if that's the case I'm not sure how to fix that as I explained unfortunatly I'm a newbie in this.

f I switch back to the backup (old file version before the issue) everything works, but of course without the --nosmooth

Any suggestions to fix that issue on my side so I can test the new file version?
Thanks ahead.

[prajwalkr]

Any suggestions to fix that issue on my side so I can test the new file version?

download the latest version once more. or the whole repo once more.

[AlonDan]

GOOD NEWS!
I just downloaded the latest file version again and did a test again, from the SAME video as the original issue, now it's STEADY as it should be, probably need to test on more complicated video situations but from the current one it's IMPRESSIVE!

Great job, thank you for the super quick fix!
here is the current version in case anyone would like to see the difference with --nosmooth
The mouth now following every single frame!

DOWNLOAD same video using --nosmooth as (zip file)

result_video (nosmooth enabled).zip

Suggestion:
Please consider to mention the use of --nosmooththis on the main GitHub in case people would like to use it.

Same Frame with --nosmooth and without:

[prajwalkr]

Please consider to mention the use of --nosmooththis on the main GitHub in case people would like to use it.

Will do it right away!

[dipam7]

I'm seeing weird artifacts near the lips for some videos even after using --nosmooth. Is this because I am using high res videos?

[prajwalkr]

It’s hard to say without seeing the video. But, yes, high res videos and faces can produce artefacts.