pose detection

[de-code]

resolves #50

ported pose detection

[de-code]

This is currently ported the multi-pose detection from JS. It doesn't seem to work very well yet. Perhaps some of the keypoint mapping is incorrect.

[gmontamat]

Hi @de-code , I'd like to help you debug this issue since I'd like to perform instance segmentation using this lib (#70). When you say it doesn't seem to work very well, what do you mean? incorrect keypoints, lack of keypoints being detected?

[de-code]

Hi @de-code , I'd like to help you debug this issue since I'd like to perform instance segmentation using this lib (#70). When you say it doesn't seem to work very well, what do you mean? incorrect keypoints, lack of keypoints being detected?

Hi @gmontamat it is all "working" to some extend and there is an example command in the updated README. You could have a look at try it with different videos or inputs.
It is not completely wrong, but it doesn't seem to work as well as I was hoping for. Therefore I am not sure whether it is useful without further polishing.
Please have a look and see whether you can make it work for your use-case.
Looking forward to your contribution.

It may also worth noting that Google keep adding things to MediaPipe. They may also add pose detection, if they haven't already.

[gmontamat]

@de-code Thanks for your response. Have you compared keypoint detection against the tfjs implementation?
My goal would be to implement the multiperson functions in python (#70) for running predictions on images. I don't think Mediapipe has such feature (human instance segmentation), though I used and modified mediapipe python lib for keypoint detection in realtime (google-ai-edge/mediapipe#2041)

[de-code]

@gmontamat to be honest I can't quite remember whether I had run the tfjs version. That would certainly help to debug it. The changes so far were mainly driven by trying to read the code.

[gmontamat]

Found something strange in
https://github.com/de-code/python-tf-bodypix/blob/pose-detection/tf_bodypix/bodypix_js_utils/build_part_with_score_queue.py#L38.

num_keypoints is not defined in the tfjs version

https://github.com/tensorflow/tfjs-models/blob/c08dae1739f8cc42502c1770ce4f314052ece2e8/body-pix/src/multi_person/build_part_with_score_queue.ts#L52

Instead, this value is obtained from the scores buffer. I assumed the last dimension of this scores buffer should be of size 17 (number of keypoints defined in https://github.com/de-code/python-tf-bodypix/blob/pose-detection/tf_bodypix/bodypix_js_utils/keypoints.py#L11) but when debugging I found out it's 24 (is it the number of part channels?

python-tf-bodypix/tf_bodypix/bodypix_js_utils/part_channels.py

Line 5 in 91820c7

PART_CHANNELS = [

)

On the other hand, the last dimension of the offsetBuffer is 34=17x2 as expected.

[gmontamat]

Ok. I think I found the bug in question @de-code
I've changed float_part_heatmaps for float_heatmaps in BodyPixResultWrapper and got good results:

diff --git a/tf_bodypix/bodypix_js_utils/build_part_with_score_queue.py b/tf_bodypix/bodypix_js_utils/build_part_with_score_queue.py
index e56babb..9be862e 100644
--- a/tf_bodypix/bodypix_js_utils/build_part_with_score_queue.py
+++ b/tf_bodypix/bodypix_js_utils/build_part_with_score_queue.py
@@ -35,9 +35,9 @@ def build_part_with_score_queue(
     score_threshold: float,
     local_maximum_radius: float,
     scores: T_ArrayLike_3D,
-    num_keypoints: int
+    # num_keypoints: int
 ) -> Deque[PartWithScore]:
-    height, width, _ = scores.shape[:3]
+    height, width, num_keypoints = scores.shape[:3]
     part_with_scores = []
 
     LOGGER.debug('num_keypoints=%s', num_keypoints)
diff --git a/tf_bodypix/bodypix_js_utils/multi_person/decode_multiple_poses.py b/tf_bodypix/bodypix_js_utils/multi_person/decode_multiple_poses.py
index d96826b..fa28434 100644
--- a/tf_bodypix/bodypix_js_utils/multi_person/decode_multiple_poses.py
+++ b/tf_bodypix/bodypix_js_utils/multi_person/decode_multiple_poses.py
@@ -66,7 +66,7 @@ def decodeMultiplePoses(
 
     queue = build_part_with_score_queue(
         scoreThreshold, kLocalMaximumRadius, scoresBuffer,
-        num_keypoints=NUM_KEYPOINTS
+        # num_keypoints=NUM_KEYPOINTS
     )
     # LOGGER.debug('queue: %s', queue)
 
diff --git a/tf_bodypix/model.py b/tf_bodypix/model.py
index 37f309d..5633235 100644
--- a/tf_bodypix/model.py
+++ b/tf_bodypix/model.py
@@ -397,8 +397,12 @@ class BodyPixModelWrapper:
         })
 
         return BodyPixResultWrapper(
-            segments_logits=tensor_map['float_segments'],
-            part_heatmap_logits=tensor_map['float_part_heatmaps'],
+            segments_logits=self.find_tensor_in_map(
+                tensor_map, 'float_segments'
+            ),
+            part_heatmap_logits=self.find_tensor_in_map(
+                tensor_map, 'float_heatmaps'
+            ),
             short_offsets=self.find_tensor_in_map(
                 tensor_map, 'float_short_offsets'
             ),

[de-code]

That is a great find. Thank you for that. I guess I added num_keypoints because it didn't match. That should have made me think twice. We will still need float_part_heatmaps for the other part segmentation.

[de-code]

If you don't mind, could you please check the updated PR?

[gmontamat]

On it. Thanks a lot. I'm testing both pose and semantic bodypart segmentation. Hopefully, I'll have time later this week to implement #70

[gmontamat]

Awesome! I was able to implement the other multi-person functions (which do instance segmentation - #70 ) on top of this branch here: https://github.com/gmontamat/python-tf-bodypix/tree/multiperson-segmentation

[de-code]

Feel free to submit a PR when you feel ready