Skip to content
@wearable-motion-capture

WearMoCap

A wearable motion capture system to estimate arm poses from a single smartwatch, or from smartwatch and smartphone data combined.

Welcome

Our wearable motion capture system provides arm-pose estimations from a single smartwatch. This allows motion capture of the human arm anytime and anywhere. Our system comprises apps to stream sensor data from wearable devices, a python package to estimate arm poses from streamed sensor data, and an optional 3D visualization using Unity.

Watch Only GIF

The associated paper to this system. New ICRA publication to be released soon 🎉

@INPROCEEDINGS{weigend2023anytime,
      title={Anytime, Anywhere: Human Arm Pose from Smartwatch Data for Ubiquitous Robot Control and Teleoperation}, 
      author={Fabian C Weigend and Shubham Sonawani and Michael Drolet and Heni Ben Amor},
      booktitle={2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)}, 
      year={2023}
}

Quick Start Guide

For more detailed instructions and advanced modes please see our wiki. As shown below, this framework features three modes. Watch Only estimates arm poses from the data of a single smartwatch. +Phone Upper Arm enables our most accurate arm pose predictions. +Phone Pocket allows to move around and change body orientation. The quick start guide gets you started with the Watch Only mode.

Modes

The Modules and Pipeline

Our wearable motion capture system is composed of modules. Each of these modules is one repository in this GitHub Organization. Namely, they are:

  • sensor-stream-apps provides the apps to stream sensor readings from wearable devices to a remote machine.
  • arm-pose-estimation opens sockets to receive streamed data from wearable devices and processes them into arm posture estimations.
  • arm-pose-visualization visualizes arm posture estimations in real time using a 3D avatar.

The figure below summarizes the data stream from smart devices to visualization. The individual components are connected through UDP streams, but Arm Pose Estimation and Arm Pose Visualization can run on the same machine. The UDP streaming from smart devices to the Arm Pose Estimation requires a local WiFi connection.

Modules

Basic Installs

First, install our sensor-stream-apps with Android Studio. Your smartwatch has to be in developer mode and has to be connected the same WiFi as the machine that you want to run arm pose estimations on. If you are unfamiliar with installing an app in developer mode, we recommend following the step-by-step instructions in our extensive wiki.

Then, clone arm-pose-estimation and install it with pip:

pip3 install /path/to/cloned/repository

For streaming data from the watch only, copy the stream_watch_only.py from the example_scripts directory. Run the script with your local IP as a parameter. For example:

python3 stream_watch_only.py 192.168.1.123

Finally, to visualize arm pose estimates, you can either download a build of the visualizer tool here, or you clone our arm-pose-visualization repository and open it in Unity. Please also see our wiki if you want to use Unity.

Pinned

  1. arm-pose-estimation arm-pose-estimation Public

    Python scripts to make arm posture predictions from streamed wearable sensor data.

    Python

  2. arm-pose-visualization arm-pose-visualization Public

    A Unity project, which opens a socket to receive arm pose estimations and then visualizes them using a 3D avatar.

    C# 3

  3. sensor-stream-apps sensor-stream-apps Public

    A WearOS and an Android app, which allow streaming IMU and other sensor outputs to a remote machine.

    Kotlin

Repositories

Showing 4 of 4 repositories
  • .github Public

    Public organization repo for documentation

    0 0 0 0 Updated Feb 11, 2024
  • sensor-stream-apps Public

    A WearOS and an Android app, which allow streaming IMU and other sensor outputs to a remote machine.

    Kotlin 0 Apache-2.0 0 0 0 Updated Feb 10, 2024
  • arm-pose-estimation Public

    Python scripts to make arm posture predictions from streamed wearable sensor data.

    Python 0 Apache-2.0 0 0 0 Updated Feb 8, 2024
  • arm-pose-visualization Public

    A Unity project, which opens a socket to receive arm pose estimations and then visualizes them using a 3D avatar.

    C# 3 Apache-2.0 0 0 0 Updated Dec 3, 2023

Top languages

Loading…

Most used topics

Loading…