DFL final face jittering problem

[iperov]

here video comparing of different face swap repos.
https://streamable.com/m8lld
as you can see DFL has problem with jittering of final face.

any ideas why it occurs?

[TooMuchFun]

I see that the jittering occurs most near the bottom left and right corners. I similar this when training faces too. I notice it most when faces are light-skinned faces, and a strong shadow or hair immediately below the jawline. My tests are mostly with LIA128 tho but maybe the two are related.

[Eleganze]

Pixel perfect tracking has been available in video editors for decades. Not sure why it is necassary to solve this problem again for faceswap purpose. Why not extract the face from tracking features instead of using face detection?? Or maybe do a combination? Locate the face in first frame - then use tracking features to ensure that the same relative pixels are extracted on the next frames. Then try to apply the actual face detection on the extracts - which then will be much easier because the image area is constant.

[TooMuchFun]

@Eleganze is there a python library that provides function? Automating the pipeline is key. Are there any open source editors that we can test?

[iperov]

Jitter is result of AE, not converter.

[Eleganze]

The way I see it, you would need pixel perfect extract - then pixel perfect reconstruction - then pixel perfect replacement. If any of these are off, you can not possibly get a result without noticeable artifacts/jittering.

[iperov]

Converter feed to AE same dst face that was trained.

[iperov]

Do you know how AE trained? Run any model train debug and see what samples feeded to AE.
What is pixel perfect you talking?

[Eleganze]

When you try to apply changes to moving footage, the changes needs to be relatively identical. Maybe I misunderstand, but I have been working a lot with Nuke and Syntheyes where you easily can replace part of a videoframe with whatever you want without visible artifacts. With pixel perfect I mean the tracking of the key features in footage you extract. The current methods used in Faceswap are done on a frame by frame basis. This means you don’t actually track and measure accuracy across the entire footage. This will cause sliding/jittering. The AE can only train on whatever you give it.

[Eleganze]

Here is example https://youtu.be/TGASWfUC8xI

[TooMuchFun]

The current methods used in Faceswap are done on a frame by frame basis. This means you don’t actually track and measure accuracy across the entire footage

Do you have an example implementation of accuracy tracking or measurement that could be added to the project? Otherwise, can you write some more on how exactly we could use these specific tools with regard to DeepFaceLab itself?

[gessyoo]

Eleganze, it would be very helpful if the face jittering issue could be addressed in python during the extraction/conversion process itself, without having to use additional software to post-process the converted frames. Not everyone who uses this repro has access to pixel perfect tracking software or the knowledge of how to use it.

[iperov]

Cannot understand why you all talking about tracking.

[iperov]

Watch video comparison https://streamable.com/m8lld
FaceSwap has jittering? NO
FaceSwap uses any advanced tracking ? NOO

[Eleganze]

See video above for an example on how tracking is done across the entire footage. If this was done the Faceswap way, the overlay grid would certainly jump around and not stick.

I could be all wrong about this, but I think its a bad idea to extract the face chips based on relatively weak facial features.(68 landmarks or so) A suggestion would be:

1. Track a 512x512 box with sub pixel accuracy across the entire footage, centered over the face. This is not face detection, but feature tracking. It is more accurate by far. I am sure Python got this.

2. Then extract the center 256x256 training chips based on the coordinates of this box, and not weak face features.

3. Run face detection inside the 256x256 chips, but find a way to make sure that the landmarks are locked relative to the 256x256 box. Run the detection phase and measure score until the landmark grid sits perfectly over the face across the entire footage - just like the video.

4. Write landmark coordinates to chip and start training.

[iperov]

-_-