Adding MediaPipeHolisticLandmarker to MediaPipe v0.10.13

[GeorgeS2019]

@j20001970
Is it feasible to upgrade to the latest MediaPipe v0.10.10 so the community could take advantage of the latest feature :

Holistic

https://chuoling.github.io/mediapipe/solutions/holistic.html

[GeorgeS2019]

The latest is v 0.10.11

The ongoing discussion

[GeorgeS2019]

@j20001970
Is there a need to coordinate upgrading to v0.10.11

Will that break existing version?

[GeorgeS2019]

https://github.com/google/mediapipe/releases/tag/v0.10.13

Attempting v0.10.11 here.
https://github.com/Malcolmnixon/GDMP/actions

The feedback to mediapipe community
google-ai-edge/mediapipe#5366

[j20001970]

@j20001970 Is there a need to coordinate upgrading to v0.10.11

Will that break existing version?

As the issue you linked, currently there are problem in using preprocessor directives within function-like macro on Windows, bumping to mediapipe v0.10.11 will fail to build Windows library without a proper fix. Since the issue has already been opened on upstream mediapipe repo, I think there is no need to take actions further, let's wait for next release tag hopefully fix the issue.

In the meantime, try not to urging people too hard on fixing certain things, different projects have different goals and priorities in mind.

[GeorgeS2019]

GDMP is now finally updated to the latest Mediapipe v0.10.13 and built successfully
https://github.com/Malcolmnixon/GDMP/tree/test-mediapipe
Great Job Malcolm

[Malcolmnixon]

I've submitted the preprocessor-fix PR to mediapipe (google-ai-edge/mediapipe#5392) and once it's in we could update to a commit, or wait for v0.10.14 which would probably be about a month out considering the average time between releases.

[GeorgeS2019]

#40 (comment)

[GeorgeS2019]

Mediapipe

https://github.com/google/mediapipe/tree/master/mediapipe/tasks/cc/vision/holistic_landmarker

GDMP

holistic_landmarker.h
holistic_landmarker.cc

Documentation

MediaPipeHolisticLandmarker

MediaPipe Holistic requires coordination between up to 8 models per frame —

• 1 pose detector,
• 1 pose landmark model,
• 3 re-crop models and
• 3 keypoint models for hands and face.

• Holisticlandmarker
• Holisticlandmarkeroptions
• Holisticlandmarkerresult

https://chuoling.github.io/mediapipe/solutions/holistic.html
https://github.com/google/mediapipe/blob/master/docs/solutions/holistic.md
https://developers.google.com/mediapipe/api/solutions/js/tasks-vision.holisticlandmarker

References

j20001970/GDMP-demo#18

[Malcolmnixon]

Mediapipe currently don't have their official C++ API for Holistic Landmarker. A PR (google-ai-edge/mediapipe#5063) has been started, although it does appear to have stalled for some number of months.

The release notes for 0.10.13 state "Make Holistic C++ graph public until we have a C++ API", so possibly we can implement the GDExtension classes as if the C++ API were present - inferring from the Holistic Landmarker PR and the APIs published for other languages (Java, Python, etc.). The implementation would hide the fact that it's using the C++ graph for now, and hopefully we can seamlessly switch over to the real API once it's published.

[GeorgeS2019]

@Malcolmnixon
Please help me here. Do we need to wait to have a working v0.10.13 dependent GDMP with the MediaPipeHolisticLandmarker ?

[Malcolmnixon]

Do we need to wait to have a working v0.10.13 dependent GDMP with the MediaPipeHolisticLandmarker

I'm not sure exactly what you're asking - GDMP is currently based on mediapipe v0.10.13 (with the necessary patches applied in the build process) and appears to work fine in my testing.

I'm suggesting the next step is to design the GDMP MediaPipeHolisticLandmarker and MediaPipeHolisticLandmarkerResult objects and their associated methods in such a way that they can be implemented using the v0.10.13 Holistic C++ graph API, but also with the intent of switching to the proper Holistic C++ API once mediapipe releases it. That way anyone writing code based on the Godot API will hopefully not experience API breakage going forwards.

[GeorgeS2019]

I'm suggesting the next step is to design the GDMP MediaPipeHolisticLandmarker and MediaPipeHolisticLandmarkerResult objects and their associated methods in such a way that they can be implemented using the v0.10.13 Holistic C++ graph API, but also with the intent of switching to the proper Holistic C++ API once mediapipe releases it. That way anyone writing code based on the Godot API will hopefully not experience API breakage going forwards.

I am not C++

Are the c++ examples of v0.10.9 helpful to achieve the design and implementation for MediaPipeHolisticLandmarker and MediaPipeHolisticLandmarkerResult?

How would you like others to help and coordinate?

[Malcolmnixon]

How would you like others to help and coordinate?

My plan (hopefully this weekend) is to familiarize myself with how GDMP maps the existing mediapipe APIs to GDExtension; and then take the unfinished Holistic Landmarker C++ API PR and apply the same mapping patterns to produce a proposed GDMP GDExtension API for the holistic landmarker with the same architectural/design choices as the existing tasks.

If that looks good then I'll try to implement that proposed GDExtension API, but using the unfinished Holistic Landmarker C++ API PR as a guide to for how to map the calls to the underlying graph API.

It would be simpler to just wait for mediapipe to release their C++ API; however that PR has stalled with no changes for 4 months, so I don't see any alternative but to work around the problem. To misquote Voltaire "Don't make great the enemy of good"!

[GeorgeS2019]

Don't make great the enemy of good"!
We shall discuss further on this :-)

KIV

https://github.com/google-ai-edge/mediapipe/releases/tag/v0.10.14

Is there something needed to be there but not yet?

[GeorgeS2019]

MediaPipe v0.10.14 attempted

@Malcolmnixon
@j20001970
homuler/MediaPipeUnityPlugin#1200