Embodiment

Connection to tracking devices

Kinect V2 Connector

I developed a Kinect Connector using this API that is accessing the data, converting it into the correct the unit and coordinate system for Unreal and is accessible from Unreal Blueprints, so that it can be easily used. After first testing performance and stability are best with this approach.

Basically the connector pulls data of the different joints in a similar way as a VRPN connector does. Kinect v2 supports 24 different joints, newly including index finger and thumb. The pulled joint data is stored in a map that can be accessed via unreal later. Furthermore the connector already provides the data structure and update logic to save and pass data of multiple tracked users.

Alongside to the grabbing and saving of the data, the connector also does some pre-calculations for easier use. It transforms the vector from Kinect V2's coordinate system into the Unreal coordinate system and multiplies all the values by 100, since the raw data is coming in meters while Unreal needs centimeters (Unreal Units are now equivalent to cm). Furthermore the connector provides the delta between the torso's position at time t and time t-1, so that it's easy to map the genereal body movement into an application.

For the use of the data in Unreal the connector can accessed both by C++ source code and by Blueprints. The functions that are supposed to be called from Unreal application all use the special Unreal C++ types (e.g. FVector for three dimensional vectors), so that there are no compatibility problems.

One of the next step regarding the Kinect V2 connector is to provide as an Unreal Engine 4 Plugin instead of being directly coded into the framework. In this way the connector could be used in any Unreal project dynamically. Furthermore I want to setup a second repository for the connector itself, so that I can provide it to the Unreal community.

VRPN-Client

Use and mapping of tracking data

To be continued...