Tensorflow PoseNet body tracking for virtual reality
The Project [Work-in-Progress]
This is a demo created for use in my Master's Thesis. The thesis focuses on analyzing the social VR landscape to better understand how to make co-located VR experiences more collaborative and social.
This purpose of this demo is to provide a way for mobile VR users to engage in VR experiences together simply by placing their phones in a low-cost VR device like Google cardboard. By using phone camera-based body tracking, VR users do not have to instrument an environment to get realistic body representation in the virtual environment.
We will use this demo as a part of a user study to discover if body representation in co-located social VR experiences encourages collaboration and increases feelings of togetherness. Scenarios will be developed to study multiple co-located VR participants and co-located VR + non-VR participants.
Additional work to be done
- Use additional sensor or marker to identify z coordinate.
- Animate a 3D model with poses.
- Add facial expression tracking of both VR and non-VR participants.
- Provide multi-user support.
- Prototype will be configurable to support multiple study activities.
- Tensorflow.js PoseNet - In browser human body pose estimation model.
- A-Frame - The web framework used web-based VR prototypes.
- Katy Madier - katymadier.com
Thanks to my advisor, Professor Michael Nebeling, and the University of Michigan Information Interaction Lab for supporting the work in my master's thesis.