place the pose skeleton from MediaPipe's metric landmarks - #503
Merged
Conversation
Landmarks are placed by raycasting each one onto the depth mesh. That is right when the people being detected are physically in front of you, since the ray lands on their actual body. It is wrong when the camera shows someone who is not part of the depth scene, which is exactly what a webcam feed on the desktop simulator is: every ray hits the surrounding room geometry instead, and the skeleton ends up smeared across the walls and floor. Turning projection off places landmarks along the view ray at a fixed distance instead, which keeps the body together. Defaults to on, so device behaviour is unchanged.
Landmarks were always positioned by raycasting the depth mesh and, when that missed, by back-projecting through the camera. Neither works when the detected person is not part of the scene being sensed, as on desktop with a webcam: the rays hit the surrounding room and smear the body across it, and the back-projection uses device intrinsics that do not match the webcam's aspect, stretching the result. MediaPipe already returns world landmarks in metres relative to the hips. Those are independent of the camera and of whether a joint is in frame, so when depth projection is off they are used to place a correctly proportioned skeleton in front of the viewer. The raw metric position is also exposed on each landmark, since it was being computed and then discarded.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
pose landmarks are positioned by raycasting the depth mesh, and when that misses, by back-projecting through the camera. both assume the detected person is part of the scene being sensed. that holds on a headset. it doesn't on desktop with a webcam, where you're sitting outside the simulated room: the rays hit the walls and furniture and smear the skeleton across them, and the back-projection uses device intrinsics that don't match a 16:9 webcam so what survives comes out stretched.
adds
world.humans.useDepthProjection(default true, so nothing changes for existing apps or on device). with it off, landmarks come from mediapipe's world landmarks instead, which are metres from the hip centre and independent of both the camera and whether a joint is even in frame, so the body keeps its real proportions.also puts
metricPositionon each landmark. it was already being computed and everything butzthrown away.