Accuracy difference in tfjs hand pose official demo and this app

[delebash]

Windows 10 Professional
Intel i9 9900k
Graphics Intel UHD 630

It seems that accuracy is not as good and also there is a performance lag between the official hand pose demo and this application. This is just my observation from testing the demo and and your human app. The distance from the camera remained constant between the two applications. Am I doing something wrong?

Video demonstration of difference.

hand-pose.zip

Thanks.

[vladmandic]

The official version is hard-coded for a single hand detection.

I've modified it for variable number of hands, but I suspect I introduced a bug somewhere as well - I'll take a closer look.

[vladmandic]

i found couple of issues in hand bounding box detection that was causing it to flicker in and out and thus impact further object detection inside that box - new version is now published

there is still room for improvement (better caching and cache invalidation to remove detection delays when hand comes in focus) when using multi-hand detection, but it should be better than it was

if you're comparing performance to original model, best to set config.hand.maxHands = 1 to set level playing field

[delebash]

Thank you for the update.

[vladmandic]

quick q - can you confirm the accuracy is now on-par?

[delebash]

I just checked your latest code and it may be a little better but not on-par with original. Thanks!

[vladmandic]

thanks for trying.

few quick questions:

• what is not on-par
  performance or quality?
• did you try reducing maxhands to 1?
  you can just drag maxobjects in UI and it will change config of all underlying models
• original model uses default values for scoreThreshold=0.5, iouThreshold = 0.3, minConfidence = 0.8
  my values are not the same - how does model perform if you set the same values?
  it's configurable in ui

[delebash]

The detection part does not work as well. For example with my hand in the same position and distance from the camera the original code will detect my hand and finger positions such as a peace sign. However on the human app my hand is intermittently detected and my fingers are not detected well. I cannot change my hand from open to peace sign and have it detect my fingers. I did set MAX OBJECTS to 1 by config and by dragging. The video attached should show the difference if a handpose is detected correctly or not. I adjusted the values to match those of the original demo detectionConfidence = 0.8 iouThreshold = 0.3 scoreThreshold = 0.75 MAX OBJECTS = 1. On the original demo maxContinuousChecks = infinity but hand app has SKIP FRAMES = 10 I left this value at 10.

[vladmandic]

note that original still has scoreThreshold = 0.75 in docs, but code has been updated to scoreThreshold = 0.5

i'll run deeper compare on my side

[vladmandic]

i've spend some time analyzing this - would appreciate your feedback:

handpose uses two models - first to detect bounding box of a hand and second to detect landmarks (finger points)

• landmark detection model if really good
  but requires very specific bounded input to work
• box detection model is horrible
  it only works on near-perfect hand detection plus it throws tons of false-positives with high confidence

so original implementation uses box detector to find a hand and once it does,
it locks in on first high confidence result and caches is as much as it can
(as long as landmark detection works it doesn't re-run box detector)
that's why landmark detection continues to work with one or two fingers, etc. although box detector would fail if it were to run

but if i want to have multi-hand support, i need to invalidate cache periodically to check if there are new hands in the frame
which backfires if existing hand is now below threshold so landmark detection has nothing to work with

i've improved caching and cache invalidation significantly
plus there is separate handler depending on maxObjects value
it's still not perfect, but i feel it's much better

new version is now published,
any comments are welcome

[delebash]

It is working great for 1 hand. I would say it is on par or better than the original. Thanks

[vladmandic]

thanks for the feedback

i'd use a better hand detector model, but size is the issue - this is the only one small enough as i don't want the library size to be unmanageable

closing as resolved