When using Mediapipe Hands, it seems that mediapipe does not want to return landmarks about which it is unsure. This makes the tool barely useable for things like gesture recognition. Why don't you allow a setting in which ALL landmark guesses are returned (possible with a confidence value) so the end user can decide what to keep, what to correct and what to discard?

[jdambre]

Please make sure that this is a feature request.

System information (Please provide as much relevant information as possible)

• MediaPipe Solution (you are using):
• Programming language : C++/typescript/Python/Objective C/Android Java
• Are you willing to contribute it (Yes/No):

Describe the feature and the current behavior/state:

Will this change the current api? How?

Who will benefit with this feature?

Please specify the use cases for this feature:

Any Other info:

[kuaashish]

Hi @jdambre,
Thanks for raising this feature request! This looks like an interesting feature request. Would you also please share a use-case where this can be useful? Thank you!

[google-ml-butler]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you.

[jdambre]

Hi @kuaashish: the fact that currently, no hand keypoints are returned very often currently makes MP unuseable 'in the wild' for tasks related to detailed recognition of hand movements.
This occurs mostly for fast hand movements or for occlusions, but also sometimes for no apparent reason when frames seem perfectly clear to us. This suggests that your current approach to decide which hand keypoints are 'not good enough' to return is not quite robust. In addition, in the case of occlusions, we could still use the visible keypoints in our models (e.g., swapping back to a 2D model instead of a 3D one).

If we would have all keypoints, we can make our own cutoff decisions, train our models robust against wrong predictions or train a model to correct them. So being able to turn off the 'cutoff' to alway return all keypoints is in fact our first feature request.

But since I assume MediaPipe uses an internal uncertainty measure to decide when NOT to return hands, this can just as well be given as an output, which would allow us to set our own threshold, treat inaccurate samples differently, or develop a a more targeted approach in making our models more robust.

[kuaashish]

Hi @bazarevsky,
Could you please look into this feature request? Thank you!

[khanhlvg]

The new MediaPipe now supports Gesture Recognition out of the box. Can you try it out and see if it fit your use case?
https://developers.google.com/mediapipe/solutions/vision/gesture_recognizer

[kuaashish]

Hello @jdambre,

Please go through the above comment. Thank you

[jdambre]

WE DO NOT NEED out of the box gesture recognition, we would very much appreciate keypoint confidences in order to be able to catch problematic cases more adequately and develop our own applications on top of mediapipe. As it stands now and since we receive no adequate reaction to any of our requests (there is at least one other thread related to failure cases), we see no other option than to develop our own keypoint extractor on top of more recent state-of-the-art tools!

[kuaashish]

@jdambre,

MP Tasks allows the user to configure detection thresholds.

[kuaashish]

@jdambre,

You're using an old version of MediaPipe's hand solution, and we no longer support it. Please check our guide for instructions on upgrading to the new Hands API, which also allows you to configure detection thresholds. We suggest switching to our new Hand Task API, as explained in the documentation here.

Additionally, you can find a code example in the same resource for your reference. If you face any issues with the upgraded API, please let us know. Thank you.

[m-decoster]

@kuaashish

It is true that is it possible to set a threshold on the palm detection, but this is different from obtaining confidence values. If I set a threshold of for example 0.1 and get 10 hands back from 15 images, I have very little information on the confidence of MediaPipe about these hands (only that the confidence should be higher than 0.1).

As far as I can tell from the documentation in the new solution, this confidence value remains an internal value in the Python API, which only returns handedness, 3D coordinates, and 3D world coordinates.

What @jdambre is requesting, is that the internal confidence value which is compared with the user-set threshold is also returned. I am referring here to this comment which mentions that you might as well return the internal confidence value to allow more flexibility in the API.

I hope this is clear.

[kuaashish]

Hi @m-decoster,

Thank you for providing additional information about this issue. We have marked it as a feature request and sharing it with the team. The team will prioritise the work based on our discussion.