1. Overview
Our WearMoCap (wearable motion capture) system is composed of modules. Each of these modules is one repository in the GitHub Organization and one chapter in this wiki.
- Sensor-stream-apps provides the apps to stream sensor readings from wearable devices to a remote machine.
- Arm-pose-estimation opens sockets to receive streamed data from wearable devices and processes them into arm posture estimations.
- Arm-pose-visualization visualizes arm posture estimations in real time using a 3D avatar.
As shown below, this framework features three modes. The Watch Only mode uses only a single smartwatch for arm pose predictions. The +Phone Upper Arm mode provides high-precision arm pose estimates. For this mode you have to use a fitness strap or something similar to strap a smartphone to our upper arm. Finally, the +Phone Pocket mode allows the most freedom. In this mode you can place a connected smartphone in any pocket. The phone then tracks your body orientation, which allows for arm pose estimates while you move around or change orientation. Please find more detailed descriptions in Section 2.
The figure below summarizes the data stream from smart devices to visualization. The individual components are connected through UDP streams, but Arm Pose Estimation and Arm Pose Visualization can run on the same machine. The UDP streaming from smart devices to the Arm Pose Estimation requires a local WiFi connection.
This section outlines the most basic steps to get started with our system. For more detailed instructions and advanced modes (like phone and watch tracking or voice commands), please see more detailed sections in this wiki.
First, install our sensor-stream-apps with Android Studio. Your smartwatch has to be in developer mode and has to be connected the same WiFi as the machine that you want to run arm pose estimations on. If you are unfamiliar with installing an app in developer mode, we recommend following the step-by-step instructions in Section 2.
Then, clone arm-pose-estimation and install it with pip:
pip3 install /path/to/cloned/repository
For streaming data from the watch only, copy the watch_only.py from the example_scripts directory. Run the script with your local IP as a parameter. For example:
python3 watch_only.py 192.168.1.123
Finally, to visualize arm pose estimates, you can either download a build of the visualizer tool here, or you clone our arm-pose-visualization repository and open it in Unity. Please also see our Section 4 if you want to use Unity.