2. Sensor Stream Apps
You can find the sensor stream apps in the sensor-stream-apps repository.
The repository contains two apps for smart devices. One Android app for smartphones and one Wear OS app for smartwatches. Together with arm-pose-estimation, these apps enable data transfer from smart devices to a remote system through UDP streams via a local network.
The apps are at least compatible with Android 11.0 (Watch App) and Android 10.0 (Phone App). They have been tested on:
| Watch App | Phone App |
|---|---|
| Samsung Galaxy Watch 5 (SM-R900 and SM-R910) | Samsung Galaxy A23 5G |
| Fossil Gen 6 Smartwatch (DW13F2) | OnePlus Nord N100 (BE2011) |
| TCL 40xl (T608M) |
Disclaimer: We have not deployed our apps to the Google Play Store yet. For now, you will have to install the apps manually by switching your device to developer mode. While the other modules of our motion capture system are much quicker to install, this section takes a few more steps.
Setting up our apps includes installing Android Studio, connecting the smartwatch or smartphone to it, and building our app to transfer it to the respective device.
Begin by installing android studio following these instructions. Then, clone our sensor-stream-apps repository and open it in android studio. Simply follow the prompts displayed in android studio as:
Click Open → Navigate to folder containing the repo → Select Open
Note: This is a one-time process required to get the apps up and running. Once installed, you can start our apps without Android Studio directly from your device.
Connect the smartwatch to the same local wifi which your computer with android studio is connected to.
To activate the developer mode on the smartwatch, follow these steps:
Settings → About watch → Software info → Tap software version / build number 9 times
A popup will show that developer mode has been activated.
Then, go back to the settings overview, and check out the new Developer options. Enable wireless debugging and select pair new device (on smartwatch) by following the instruction shared on Android Studio for Pair Devices Using Wi-Fi inside the Device Manager. Follow the instructions to pair with a code. If successful, the smartwatch will show up in the physical devices tab.
If your smartwatch allows for this, you can alternatively plug in the smartwatch to your computer via the charging cable. Select Physical inside the Device Manager and choose your device. _Note: Ensure debugging options are enabled. _
Now, in android studio, make sure the following 2 selections have been made. First, make sure watch is selected, next to the hammer (make project) icon.
Second, select the smartwatch model/name from the top dropdown menu. This menu is on the right of the previous menu selected and next to the run icon (represented as a green arrow/triangle). Now, click the run icon to build and install the app. After a minute or so, the app should start itself on the watch.
Once completed, the app is installed on the watch and you will be able to start it from the app overview without building it anew. It is the icon at the very bottom with the green android head.
Ensure all prompted permissions are granted once you open the app for the first time. This is especially important in smooth functioning of the app.
Connect the smartphone using a wired connection to the computer with android studio.
To activate the developer mode on the smartphone, follow these steps:
Settings → About device → Software information → Tap “Build number” field 7 times
A popup will show that developer mode has been activated. For reference, you may visit this website for a pictorial tutorial or more information.
Go back to the settings overview, and check out the new Developer Options entry. Enable USB debugging.
In android studio, open the Device Manager, click on Physical, and select your smartphone. If successful, the smartphone will show up in the Physical Devices tab.
Now, in android studio, make sure the following 2 selections have been made. First, make sure phone is selected, next to the hammer (make project) icon.
Second, select the smartphone model/name from the top dropdown menu. This menu is on the left of the green arrow/triangle). Now, click the run icon. Once done, the app should show on the homescreen or the app drawer (based on your settings).
Ensure all prompted permissions are granted once you open the app for the first time. This is especially important in smooth functioning of the app.
This section informs you about how to use the installed apps and how to troubleshoot.
After launching the watch app, it will prompt you with a mode selection. The modes Watch Only, +Phone Upper Arm, +Phone Pocket are depicted below.
-
Watch Only This mode uses only smartwatch data to estimate arm poses. Only select the this mode if no phone is connected. Also, make sure the watch is connected to a local WiFi network. If the watch is still connected to a phone via Bluetooth, the watch might show that it is streaming data, but the data does not arrive on the remote machine. For now, this mode only works with the left arm.
-
+Phone Upper Arm This mode makes the most precise arm pose estimates. We connect the smartwatch to a smartphone and strap the phone to your upper arm with a fitness strap. The watch will then stream its data to the phone, and the phone adds its own IMU data and broadcasts it to the connected computer via WiFi. This mode works for the left and right arm. For more instructions, see the details in the Phone App section.
-
+Phone Pocket We introduced this mode in our ICRA Paper. This mode allows the most movement freedom. Also here the smartwatch has to be connected to a smartphone. You can then put your phone in any pocket and it will track your orientation while the watch tracks your arm pose. This mode only works for the left arm. For more instructions see the details in the Phone App section.
In +Phone Upper Arm Mode we ask the user to complete a calibration procedure. In Watch Only and +Phone Pocket the calibration happens automatically from the start pose as depicted below.
If you selected the modes +Phone Upper Arm, please calibrate the app by pressing Recalibrate. It’s a two-step process:
-
When you press start, hold the watch in front of your chest at a 90 degree angle, parallel to your hip. Wait until the watch vibrates.
-
Now stretch the arm forward, perpendicular to the hip. Make sure you hold the watch level to the ground. The watch stops vibrating once you found the right position (as in the figure below in the middle). In +Phone Upper Arm, please remain in this position for just a bit longer until the phone finished calibrating and the watch vibrated a final time.
Note: If the watch keeps vibrating, it is to indicate that your watch face/wrist is not perpendicular and/or not level (directly facing upwards).
In the modes Watch Only or +Phone Upper Arm, we assume a fixed body orientation. So, if you want to use the watch while facing into another direction, just restart the IMU streaming or press Recalibrate after changing your body orientation.
The smartphone app becomes relevant when you selected the modes +Phone Upper Arm or +Phone Pocket on your watch. Make sure the watch is connected to the smartphone via Bluetooth. The watch will then stream its data to the phone, and the phone adds its own IMU data and broadcasts the combined data to the connected computer via WiFi.
After launching the app, you need to update the broadcast IP and hand mode (left or right) by pressing the Set IP and Hand Mode button.
At the current state, the right-hand mode only works for the +Phone Upper Arm setting.
In the mode +Phone Upper Arm the smartphone needs to be strapped to the upper arm, on the same arm’s triceps as your watch. With your arms straight down.
In the mode +Phone Pocket you can place the phone in any pocket, then start the IMU streaming from the watch. The phone will track your changes in body orientation.
Some common error cases to check:
- The connection tab at the top of the screen displays active connection with the smartwatch. If you see “Awaiting Watch App” even when the smartwatch app is active (and +Phone Upper Arm mode is selected), there may be a few possible issues:
- The watch may not be properly connected/paired with the phone
- The apps needs a restart
- The smartphone app and smartwatch app were downloaded to the system by different users (signatures). Therefore, ensure both the smartphone and smartwatch apps are installed by the same system and user in Android Studio.