-
Notifications
You must be signed in to change notification settings - Fork 19
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Multiple poses not outputting #62
Comments
Hey @marbel9 I was just tinkering with the multi pose stuff this week, so good timing on your question! The short version is multi-pose tracking in MediaPipe isn't very good right now, and I'd suggest you use something else to get good results. There is an older body tracking plugin Torin made that does an ok job of multi pose, or if you are running Windows and have an Nvidia RTX GPU the built in Body Track CHOP can do multi pose. That being said, I have made some changes that dramatically smooth out the jittery pose tracking and add a second person to the pose_tracking output CHOP that will be in the next release, and hopefully Google come up with some better pose tracking models down the road that we can incorporate too. I should note that even with 2 outputs in the yet-to-be-released version, it's still not a true multi person tracker. The true multi person trackers include additional data that tell you which person is in each pose object, which greatly helps with those scenarios you described where pose0 data jumps between different people from frame to frame. For example if you have 2 people in the frame, a multi pose tracker will assign each person an ID (let's say person A and person B). In frame 1 person A is in pose object 0, and person B is in pose object 1. In the next frame of video, for a whole bunch of reasons, person B may be the first one detected so their pose information goes into the pose 0 object, and person As data goes into the pose 1 object. If you are only looking at pose 0 in TD that will appear as if person A and B suddenly swapped places. A multi pose tracker will include an additional ID tag to say "pose 0 object is person A in this frame" so you can track the same person frame-to-frame and not have them bounce around. Anyhoo, yes the multi pose will be getting a little better in the future, but other options do a better job for now if you need multi pose tracking. |
Hi Dom, thanks for the info! Also great to hear you are still working on this project :) Will the next release work on newer versions of TD as well? |
Yeah, we keep chipping away at improving things and adding new features as/when Google add them to MediaPipe! And yup, we're making sure the plugin runs on 2022 and 2023 for now. At some point we might have to drop support for older versions but for now you're good! |
FYI the latest release now exports multiple poses, but I don't think you want to use it. There's no overlap checking right now, so you could get 2 poses detected for the same person... which is less than ideal. I added in some smoothing so it's slightly less jittery, but it's not what I'd call... good. Hopefully Google tidy it up in a future release of MediaPipe, but for now the capability to handle multiple poses is in our plugin and it's performance will improve as/when the underlying MediaPipe does :) |
I have the latest version of the release but cannot for the life of me figure out what I need to do to get the pose 2 data. I realize that it's going to be Not Great but I still would like to get it working. If there's documentation for this somewhere I haven't been able to find it. My problem is essentially the same as the original issue - I have poses dialed up to 2 and can see 2 poses in the pose_tracking but the output is only one set of coordinates. |
Ok so 2 poses in the tracking is good progress! In order to make the multi-pose data backwards compatible, and to make it easier to match with unknown numbers of poses, I added each additional pose after the first one as another sample to the normalised_results CHOP. That means if you only have 1 pose, you get a channel per keypoint with it's confidence value, same as before. Any additional poses that are detected are added samples to the output CHOP. You can tell this is happening when the viewport switches from the grey single-channel view to the black and multi-coloured line view to indicate multi-sample data. You can also right-click and see the Info on the CHOP to see how many samples ( If you want to have each pose as it's own channel, you can use a Shuffle CHOP to split every sample into a new channel ( A more scalable way to handle things is to keep the pose data in the multi-sample domain. It has a bit of a learning curve to get your head around, but it works well. As an example, let's say you want to count "how many people have their hands above their heads"
If you find any CHOPs that are turning multi-sample data into single channel, make sure I've attached a tox from another project I'm working on that I think will show you what I'm talking about. It's looking for how many people have:
The tox is for TD 2023 and hopefully gives you a starting point to play with :) |
Wow thank you for the very thorough explanation! |
I'm sure that this isn't a real issue but I can't figure out how multiple poses are meant to work. I have dialed up the pose count from 1 and while media pipe seems to track multiple poses just fine, the output only gives me coordinates for 1 pose at a time, usually flipping back and forth between my pose and the person's next to me.
How is this function meant to work?
Also love these toxs, thank you!
The text was updated successfully, but these errors were encountered: