Tracking

General

The main tracking device is Microsoft Kinect v2. As a general experience I made in the early stages of this project, I can report that Kinect v2 is very picky in terms of USB controllers.

Firstly the device only runs on USB 3.0 Ports. Secondly the USB 3.0 has to have a Chipset made by Intel or Renesas, other manufacturers (i.e. VIA) are not supported. For my home setup, I chose this USB 3.0 controller: http://goo.gl/6yULCG

My PCs is based on the Socket 775 plattform and therefore has no native USB 3.0 support. Furthermore the mainboard still uses the PciExpress 1 standard. That's important because being on this standard it is not enough to install the USB 3.0 controller in a PciEx x1 slot. It doesn't provide enough bandwith for the Kinect to work on it. You have to install the controller in, at least a PciEx x4 slot to make it work. The frame rat on my system is 30fps.

Right now there are three different approaches to connect the device to UE4:

1. Kinect 4 Windows v2 - Unreal Engine Plugin
2. FAAST (VRPN Server) - Kinect v2 Version
3. Kinect 4 Windows v2 - C++ API

Kinect 4 Windows (K4W) Plugin

Approch number one - the K4W-Plugin - is set up by checking out the repository (preferably on the latest branch) of the plugin on Github. As of now (Nov. 21st 2014), 4.5.0 is the current version and can be found here: https://github.com/lion03/UE4-K4W-Plugin/tree/KinectV2Plugin-4.5

Right now it is neccessary to fully compile the version of the engine that contains the plugin. So basically, what one gets from the referenced repository is a full version of UE4 containing the Kinect integration.

To get the idea of using the Kinect integration in one's own project, there's an example project that shows how the Kinect functionalities are used in the editor (the author is using blueprints in his example, so there is no example on how to access the data from the source code): http://goo.gl/dnG4zA

In the beginning of January 2015 a new version of the plugin was released. The author says it also includes a motion capturing example, so it's necessary to check if this may bring any advantage.

FAAST K4W (VRPN Server)

Approach number two - the K4W version of the Flexible Action and Articulated Skeleton Toolkit - is setup by downloading and unpacking the archive linked earlier on this page. Hereafter the executable can be started, which opens the toolkit and allows to create a connection to the Kinect SDK. Once the connection is established, the VRPN server is up and running (the toolkit is configured to provide a server by default).

A VRPN Client from within the application can access the VRPN server and pull the current tracking data. The server can be accessed via IP-Adress ("localhost" or "127.0.0.1" if you're running the server on the same machine as the client) and the tracker name -> i.e: "Tracker0@127.0.0.1", the tracker names are usally specified in the server application.

Kinect only uses one single VRPN tracker, while the limbs can be accessed by calling this tracker with an index (ranging from 0-23). Lots of other tracking system (like Vicon motion capturing systems) use a 1:1-relationship in terms of trackers and limbs (e.g. rigid body targets), so you need to access each different tracker to accquire all the data.

After several tests right now it is safe to say that the performance of the FAAST VRPN Server is too bad at the moment. If you access the skeleton data from Unreal via VRPN, the skeleton freezes for seconds.

Kinect 4 Windows v2 C++ API

The direct way to pull the data from the camera is the native C++ API. The Body Basics Example contained in the SDK page gives a good idea of how to initialize and access the Kinect.

I developed a Kinect Connector using this API that is accessing the data, converting it into the correct fromat for Unreal and is accessible from Unreal Blueprints, so that it can be easily used. After first testing performance and stability are best with this approach.