Any plan to support newly released mediapipe blendshape v2?

[sylyt62]

Do you have any plan to transfer mediapipe blendshape model to onnx/trt?

ref link: https://storage.googleapis.com/mediapipe-assets/Model%20Card%20Blendshape%20V2.pdf

[PINTO0309]

I don't see the point of converting the same model.

[sylyt62]

It is not the same. Actually landmarks cannot be used directly when rendering models. In AR/VR applications, people use blendshapes to drive character expressions, which is the feature supported by mediapipe recently.

[PINTO0309]

keijiro/FaceMeshBarracuda#24

google-ai-edge/mediapipe#3421

https://storage.googleapis.com/mediapipe-assets/face_blendshapes.tflite

[PINTO0309]

I expect you to contribute to the community, not just want information.
I am open to pull requests for sample code.

• face_blendshapes.onnx.zip
• https://github.com/PINTO0309/PINTO_model_zoo/tree/main/390_BlendshapeV2

$ sit4onnx -if face_blendshapes.onnx -oep cpu
INFO: file: face_blendshapes.onnx
INFO: providers: ['CPUExecutionProvider']
INFO: input_name.1: input_points shape: [1, 146, 2] dtype: float32
INFO: test_loop_count: 10
INFO: total elapsed time:  5.369663238525391 ms
INFO: avg elapsed time per pred:  0.5369663238525391 ms
INFO: output_name.1: output shape: [52] dtype: float32

$ sit4onnx -if face_blendshapes.onnx -oep cuda
INFO: file: face_blendshapes.onnx
INFO: providers: ['CUDAExecutionProvider', 'CPUExecutionProvider']
INFO: input_name.1: input_points shape: [1, 146, 2] dtype: float32
INFO: test_loop_count: 10
INFO: total elapsed time:  28.800249099731445 ms
INFO: avg elapsed time per pred:  2.8800249099731445 ms
INFO: output_name.1: output shape: [52] dtype: float32

$ sit4onnx -if face_blendshapes.onnx -oep tensorrt
INFO: file: face_blendshapes.onnx
INFO: providers: ['TensorrtExecutionProvider', 'CPUExecutionProvider']
INFO: input_name.1: input_points shape: [1, 146, 2] dtype: float32
INFO: test_loop_count: 10
INFO: total elapsed time:  3.676176071166992 ms
INFO: avg elapsed time per pred:  0.3676176071166992 ms
INFO: output_name.1: output shape: [52] dtype: float32

Kazam_screencast_00035_.mp4

[PINTO0309]

[PINTO0309]

https://github.com/PINTO0309/PINTO_model_zoo/tree/main/410_FaceMeshV2