facetracking - faceMask vs transformmatrix

[yrostudio]

Hi,
in you're example i'm trying to match the white faceMask to the canonical_face_model connected to the transformmatrix you put on your example.
rotation, ... seems to be ok. But the Z (zoom in / out with head distance from camera. When i'm closed to the camera the canonical_face_model is smaller and bigger when i'm far for the camera.

Any idea ?

My idea is to make a "kind of snapchat filter" with a 3D head oculter that follow the head size.

Thanks

[yrostudio]

I'm playing with mediapipe, amazing work !! Thanks to share !
In the face_tracking_examples we can clearly see that the torus is not following the Z- axis / distance/size of the head.
How could i do that ?

Thanks a lot.

[domisjustanumber]

Hey @yrostudio thanks for the kind words and glad you're having fun with the plugin!

I found the same issues you did when I was playing with the plugin and the short answer is I don't know why it happens. It feels like it's something to do with the dimensions being different. The canonical face mask is supposedly roughly real-world-size, so has a height of 20 units ~ 20 cm. The face mask that we're animating/generating in the MediaPipe tox is based on screen normalised coordinates (0-1), so has a height of something like 0.8 units.

The transformation matrix we receive from MediaPipe is supposed to map from the canonical face model to screen coordinates, but I get the feeling there's a wrong scale being used somewhere (or I've applied it incorrectly in the face_mapping tox). I'm not quite sure what it is, but if you figure it out I'd love to know and we can roll it into the next release!

[yrostudio]

Hi @domisjustanumber,
i found this discussion : google-ai-edge/mediapipe#1895
don't know if it helps.

Tracking eye size / iris ? to get a Z scale ?

I have no clue how to help :)