[BodyPix] single person detection in latest bodypix

[sowmyatv]

Hi,
When I try the bokeh segmentation effect using body-pix@1.0.0, It detects/segments the person (A) in front of the camera. If another person (B) is standing behind, away from A, B is being blurred out. If the person B comes very close to the contour of A, then person B is also getting detected. This is the preferred behaviour.

Now when I try with body-pix@2.0.0, both Person A and B are getting detected even though I am using segmentPerson API. Pls note, person B is standing much away from person A, still both are getting detected. The advantage I see with 2.0 is that the contour of the person detected is much more accurate and smoother than that in 1.0 which had a gap in the contour and the bokeh effect was missing around this gap. In 2.0, the contour is more accurate. But multiple people are getting detected. Is there any parameter I could tweak to restrict this to single person detection and use the smoother contour?

thanks
Sowmya

[tylerzhu-github]

Hi @sowmyatv,

Thx for using BodyPix 2.0 as well as 1.0. Your feedback is incredibly useful.

As you mentioned, BodyPix 2.0 has a much more accurate boundary. This is mainly due to some technical improvements including: up-sampling the segmentation logits to higher resolution before applying sigmoid to compute the per-pixel probability score. You could achieve the same improvement in 1.0 but it requires changing the source code a bit in the 1.0 version.

If you want to use BodyPix 2.0 to only blur just a subset of people (e.g. the large people), a quick way would be to use BodyPix 2.0's Multi-Person Segmentation API: https://github.com/tensorflow/tfjs-models/tree/master/body-pix#multi-person-segmentation.

This method returns an array of PersonSegmentation object. In your case it will be an array of two PersonSegmentation object: one for Person A and one for Person B.

You could then remove certain people (in your case Person B) from that array and pass the resulting array (with only 1 element: Person A) to the drawBokehEffect https://github.com/tensorflow/tfjs-models/tree/master/body-pix#bodypixdrawbokeheffect.

To automate this process for other cases (3 or more people):
Each PersonSegmentation object has a .pose field that contains the 2D coordinates (in image pixel space) of the person's 17 keypoints. They can be used to compute the smallest bounding box area for each person. The person bounding box area can then be used as a criteria to remove small people in the image.

Let me know if this helps! Look forward to what you will be building with BodyPix and happy to address any further feedbacks from you!!

Best,
Tyler

[sowmyatv]

Tyler,
Thank you, I greatly appreciate your detailed response. Following your suggestion, I was able to use the multi Person Segmentation API of 2.0 and get the PersonSegmentation object that I wanted. However even if I use the most lineant configuration - MobileNetV1 (with outputStride - 32, internalResolution - low, quantBytes - 1, multiplier - 1) there is a lot of latency compared to the version bodypix 1.0. Is there anything that I can tweak to make this faster?

As suggested by you, I would like to try the other option of tweaking the 1.0 source code. my understanding is that the estimatePersonSegmentationActivation in 1.0 needs to be tweaked to use predictForPersonSegmentation instead of predictForSegmentation and the segmentLogits need to be passed to the scaleAndCropToInputTensorShape
Can you pls let me know if there is anything else that needs to be changed in 1.0 to get a contour similar to 2.0?

thanks
Sowmya

[sowmyatv]

Hi Tyler,
Just to recall our previous discussion, I need to support only single person detection. Bodypix 1.0 works well and detects single person and blurs everyone behind, but the contour is not accurate. Bodypix 2.0 segmentPerson detects multiple people. So previously, I was using 2.0 segmentMultiPerson and filtering out one person. But this was very slow and unusable.

So as suggested by you before, I am trying to apply the bodypix 2.0 code optimizations to bodypix 1.0 to fine tune the contour around the single person detected. I am trying to mimic the 2.0 baseModel predict() in 1.0 mobilenet predict(). I understand bodypix 2.0 uses graph model against the layering model used by 1.0. I am facing errors when I use the 1.0 model (json) and try to do predict() on the graphmodel (tfconf.GraphModel). This is the error I am seeing : "The shape of dict['sub'] provided in model.execute(dict) must be [], but was [1,361,641,3]" This works well when I feed a bodypix 2.0 json model file.

My questions are:

1. In bodypix 1.0, should I use 1.0 model.json or use the 2.0 json which uses Tensorflow latest 1.0 model?
2. Is layering in bodypix 1.0 responsible for detecting a single person vs 2.0 detecting multiple poses? If yes, which part has to be tweaked here to improve the contour?

Can you pls provide pointers as to what changes I need to do to achieve my goal?

Thanks
Sowmya

[rthadur]

Will be closing this issue as we will be using latest body segmentation model for same purpose.
Please check latest body segmentation model for latest bodypix model demo https://github.com/tensorflow/tfjs-models/tree/master/body-segmentation

[google-ml-butler]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you.

[google-ml-butler]

Closing as stale. Please @mention us if this needs more attention.