Upgrade MediaPipe Hand model

[YaraAmin]

Hi
I need to upgrade MediaPipe Hand model to this one: https://google.github.io/mediapipe/solutions/hands
so will Human work well if I just converted the tflite model to bin ? If No, is there any recommended way to upgrade the hand model?

[vladmandic]

Can you elaborate why do you think that model would perform better?

Current hand model in Human is based on MediaPipe Hand model already with a lot of JS specific post-processing as model itself does not return processed results, but instead raw results that require a lot of processing (which MediaPipe solution does in a custom WASM module (that's why the publish only TFLite version) while Human does it in JS code and supports multiple backends)

You can see reference to the model at https://github.com/vladmandic/human/wiki/Models

I can check if there were any recent upgrades to the MediaPipe's ML model itself and switch it if necessary.

[YaraAmin]

Thank you for you answer
currently, I'm having a problem with the hand detection model as it can't detect the hand if the fist is closed or at this position https://cdn-prod.medicalnewstoday.com/content/images/articles/324/324489/fist-opener.jpg
so I tried the attached MediaPipe hand model demo and it detects perfectly in the mentioned hand positions. That's why I'm thinking of replacing it.
Also, I noticed something: when I try Human demo and I'm in the same position the models take like a second to detect all features of face and body. Do you use some kind of aggregation on frames features in the preprocessing?

[vladmandic]

currently, I'm having a problem with the hand detection model as it can't detect the hand if the fist is closed

majority is due to choice of default parameters in Human.config
in general, hand detection is the least optimized model of all of models in Human and I choose to run it with lowest priority primarily due to it's impact on overall performance

(to be precise, handdetector is horrible model, handskeleton is a good one, but it expects cropped hand box and with correct rotation as input so detecting hands is what fails most of the cases and detection is slooow, so i re-use pre-existing cropped box and move it to fit hand movement as best-guess between frames)

you can set your config to:

• disable all other models so only hand model runs
• set config.human.hand with following parameters

{
    enabled: true,
    rotation: true,
    skipFrames: 10,
    minConfidence: 0.2,
    iouThreshold: 0.2,
    maxDetected: 2,
    landmarks: true,
}

when I try Human demo and I'm in the same position the models take like a second to detect all features of face and body.
Do you use some kind of aggregation on frames features in the preprocessing?

Not aggregation, but frame-change detection
If it's below threshold, Human will interpolate new positions based on cached detected boxes instead of re-running detection models

Without interpolation and significant caching, there would be no chance to run all models combined in near real-time

You can modify config.cacheSensitivity value to increase/decrease how sensitive Human is to change-frame detection
(setting it to 0 disables caching which is good for testing)

Btw, just to be clear, frame-change detection, result caching with best-guess analysis and in-between keyframes result interpolation are three different things - they work together, but they can all function independently in Human

[vladmandic]

@YaraAmin any updates?

[YaraAmin]

Thank you for your response
Unfortunately, the results didn't improve after adding your recommended configurations

[vladmandic]

I'll take a closer look, maybe I have a bug in post-processing.

[vladmandic]

I just rewrote parts of hand detector post-processing algorithm and pushed a new update on git, hand detection should be better now
still room to improve, but i think this is a good start

btw, given this resulted in a code change i'm converting this discussion into an issue
please continue the thread in #160