Proposal for new body detection models

[nicoeiris11]

Hi Vlad,

I'd like to propose the addition of new multipose body detection models to human repo. While existing models are excellent, I believe that adding new models would provide users with more options and enhance the overall usefulness of the repository.

I would like to suggest the following models:

Name	Description	Repository URL	Paper URL
HRNet	A high-resolution network for human pose detection and segmentation	https://github.com/leoxiaobin/deep-high-resolution-net.pytorch	https://arxiv.org/pdf/1902.09212.pdf
DCPose	A deep dual-consecutive network for human pose estimation	https://github.com/Pose-Group/DCPose	https://openaccess.thecvf.com/content/CVPR2021/papers/Liu_Deep_Dual_Consecutive_Network_for_Human_Pose_Estimation_CVPR_2021_paper.pdf
CenterNet	A simple, versatile and effective approach for object detection	https://github.com/xingyizhou/CenterNet	https://arxiv.org/abs/1904.08189
DeepPose	A convolutional neural network for human pose estimation	https://github.com/asanakoy/deeppose_tf/tree/tensorflow_r0.12	https://arxiv.org/abs/1312.4659
DensePose	Dense human pose estimation in the wild	https://github.com/facebookresearch/DensePose	https://arxiv.org/abs/1802.00434

I believe that adding these models to your repository would benefit your users and make the repository more comprehensive.

Thank you for your consideration. I look forward to your feedback.

Thank you so much!

[vladmandic]

• HRNet, DCPose, CenterNet as massive (smallest is 100+MB, going all the way up to 800+MB for some variations) and even with most aggressive conversion, they are not suitable for in-browser execution, so that's a no-go.
• DeepPose is very old (over 7 years) and pretty much obsolete by now. Plus download links are broken.
• DensePose was converted to be part of Detectron2 framework and porting entire framework is far beyond my scope.

Note that currently largest model supported by human is blazepose-heavy which is ~26MB - and that's pretty much just an experiment. All actually supported models are below 15MB and any model that I might add in the future should be below that threshold.
(note that original model may be larger if it can be successfully quantized without loss of functionality - for example, 30mb fp32 model may be converted to 15mb fp16 model).

btw, if you're looking for state-of-the-art body pose detection, my favorite model is https://istvansarandi.com/dozens/
i've used it extensively in the past and done some in-browser 3d visualization, take a look at https://motioniq-ai.github.io/demo/prototype.html

[nicoeiris11]

Thank you so much for reviewing my proposal.
Totally understand these ones are big models for browser exec.

Regarding https://istvansarandi.com/dozens, looks very interesting and promising, but this one is not intended to be used in the browser, right? Or is it possible to have it integrated into human as an alternative body model? I need real-time accurate multi-body detection in the browser.

[vladmandic]

no, its not, i just noted as alternative to models you've listed - but just like those, its also waaay too large to run in browser.
i'm not aware of any better multi-pose models that would work in browser.

but if you have a good gpu, maaaybe you can try an alternative workflow?
its a bit of a pain and performance is not going to be ideal, but it would work for any number of people and you can use any high-precision single pose model.

• have two instances of human created
  one with only object detection enabled (uses mb3-centernet by default which is pretty good) and
  one with only pose detection enabled (highest precision model would be blazepose-heavy)
• run detection on input using first instance
• crop results based on detected people
• run second detection as a loop on each cropped image using second instance
  and rescale of each result so coordinates are relative to original image, not cropped one

[nicoeiris11]

After experimenting with a second human instance, I see FPS decreases by ~40% which is very bad for my analytics pipeline.

Fixed movenet-multipose seems to be very promising and hope this can be my final body model.
One of my latest requirements includes counting and tracking people, that's why I need to switch to a multi-person body model.

• In addition, I need to process how long people stayed in front of the camera, which means counting the seconds of tracked people. Any ideas for this requirement post-proc?

• I'm aware human implements tracking, but I'm not sure if result.body[X].id refers to the tracked body or just an ID. For example, if we have people "A" and "B" walking in front of the camera, result.body[X].id will be always the same for person "A" and "B"? Or is there any other identification field for tracking purposes?

[vladmandic]

id in result.body is just a detection id, cannot be used to make a logical connection between a specific person and a detection.
and results of detection do not always have to be in the same order. so yes, its an issue.

one way to handle it is to assume that people behave according to 'least action' principle.
for example, for each person get some coordinate and store it.
then in the next frame compare which person in new result has smallest distance to stored coordinate and assume that is the correct person. and then update the coordinate so it can be used again in the next frame.
that is an easy algorithm to implement.
i've used that concept before and it works pretty reliably except when people walk in front of each other.

but if you have overlapping people (walking in front of each other, etc.), then you'd need to track coordinate as well as direction of movement and find closest person in that general direction only (but allow for some angle). that would solve it for most cases except when people walk straight towards camera and then overlap.

then there is a problem of "cache expiration" - what if one person exits frame on the left and another enters on the right?
number of people in the frame is the same, but that's clearly not the same person anymore. so there has to be some threshold so distance check invalidates a person if distance is too high and treats it as a new person.

also, which coordinate to store? i'd think the best would be to calculate center of face.

regarding tracking how long they stay in front of camera? to do that reliably you'd need to solve first problem first.
and then its trivial, simply add a timer each time person is first detected and reset it when person is either no longer detected or appears too far, so its likely a different person.

sorry for not-a-short-answer, probably not what you were hoping for...

[nicoeiris11]

Hi Vlad!

First of all, the last thing you should do is apologize for this precise and excellent answer you gave me.
This is the kind of information I'm constantly looking for from repo owners, and this is why I'll keep sponsoring your great work!!

I totally agree with you that first I need to solve identify people among different frames, and after that implement counting the seconds for each person.

In my specific case scenario, occlusion (overlapping) is something very common since my system interacts with pedestrians walking in the street.

I'm aware of 2 tracking algorithms, bounding box (what you mentioned) and keypoint tracking. Do you have a formed opinion about these ones? From what I know keypoint tracking favors accuracy and bbox tracking favors performance. Need to check it though.

Looking into different solutions, I found that tfjs-pose implements tracking besides detection:

• Bbox tracking
• Keypoint tracking

Do you think I could migrate body detection to tfjs implementation while still using human for face, etc? Or even reuse this tfjs tracking implementations as body post-proc?

Again, thank you for your detailed answers 🙏

[vladmandic]

i've taken a look at tfjs-pose tracking implementation and IMO it doesn't handle the most important case - if two objects overlap, you'll always temporarily end up with just one object detected while second object disappears - which one is it? first or second? then second object re-appears and how does it associate it with object id? it doesn't.

any real world solution would need to maintain a temporal map of objects so they can be re-attached.

second thing is that body detection is nowehere near as reliable for busy frames, so it will constantly update.
its much better to base tracking on well-known detectors, either object detection or face detection (both are faaar more reliable than body detection) and then use those results to attach body results.

and human already has "attaching" part solved (there is a human.person getter that does attaching of all detected face/body/hands so they belong to same person). whats missing is actual tracker so person id can be maintained consistently.

but its a nice problem to solve.
if this is not time-critical, i'd say give me some time, i'd love to try to crack it, just busy with a different project at the moment.

[nicoeiris11]

Indeed is a very interesting problem to solve and even more dealing with occlusion.

Of course, take a look and please let me know if this is something that could be solved by human or if I should implement it myself.

Regarding human.person, are you sure we should rely on this for tracking purposes? I just tested a detection with myself in front of the camera and noticed that human.person is missing the body even though is detected. Can you please confirm I didn't misunderstand the purpose of this prop? Here is a screenshot of the result:

[vladmandic]

i haven't updated human.person getter in a while, so likely a bug due to a lot of new code added to body detection and person getter does not parse results properly. should be an easy fix.

[nicoeiris11]

Thank you for taking a look into it, please lmk once it's fixed.

[vladmandic]

i'm thinking of first implementing object-tracking as a standalone repo and then integrate it as a module to human.

[vladmandic]

ideally, i'd like to find implementation of one of those, but using a lightweight "head" such as mb3 instead of heavy ones used in reference implementations. i don't have time to do full implementation as that would require retraining from scratch.

if i cant find it, i will have to settle for algorithmic solution instead of ml one. i pretty much know how to write it, but i'd like to avoid that path.

[nicoeiris11]

Totally understand, which is the priority of this work compared to other projects/tasks? (just to have a sense of when users can enable/disable tracking from human config).

In case you don't find a lightweight ML implementation, is it possible to first have the algorithmic approach and then resume the research related to the ML approach?

Please let me know if I can help you with any task to speed up the development/release of this new feature.
Thank you again Vlad!

[nicoeiris11]

Hi Vlad! I'd really appreciate it if you let me know via this thread about any progress made for the tracking feature.
Thanks!

[vladmandic]

hi. i'm sorry, i just didn't have time to decidate for this - my current target ~2 weeks.

[nicoeiris11]

Thank you so much for your response, I really appreciate it.

[nicoeiris11]

Hi Vlad! I see you added some commits for the tracking feature. I'd like to know if this is the final version or if you're still working on it.
Thanks!

[vladmandic]

its a very early draft :)

[nicoeiris11]

Hi Vlad! Did you have any chance to move forward with this feature development?