Tutorial: Making a Module using Handtracking

Introduction

This is a tutorial to walk through building your first module using handtracking. This document will cover the basics of how to set up your module class, accessing the handtracker driver, and getting hand data.

Initial Setup

If you haven't yet made your project, see [Creating Your First Module] (https://github.com/ColoradoSchoolOfMines/interface_sdk/wiki/Tutorial:-The-Undisputed-Step-by-Step-Guide-to-Developing-with-the-Interface-SDK#3-creating-your-first-module) to get started. Make sure that you've defined a proper package name and made a class that extends a module.

Writing the First Lines

Your module should follow the format specified by the module type you're extending. Consult Extending Modules for more information on how to start your specific module type.

Using Hand Tracking

In order to use hand tracking, you will need to add an input tag to your module manifest that looks like this:

<input input-type="handtracking" />

This specifies that your module will need access to hand tracking, and this signals the [HardwareManager] (https://github.com/ColoradoSchoolOfMines/interface_sdk/wiki/API:-Hardware-Management) to give you permission to those devices.

You will also need to add a HardwareManager to your variables, and instantiate it like this:

private HardwareManager hardwareManager;
...
try {
	hardwareManager = HardwareManager.getInstance();
} catch (HardwareManagerManifestException e) {
	e.printStackTrace();
}

To get the driver itself, you will need to create a HandTrackerInterface driver. The simple way to get the driver is to do this:

private HandTrackerInterface handDriver;
...
try {
	handDriver = (HandTrackerInterface) hardwareManager.getInitialDriver("handtracking");	
} catch (BadFunctionalityRequestException e) {
	e.printStackTrace();
} catch (InvalidConfigurationFileException e) {
	e.printStackTrace();
} catch (UnknownDriverRequest e) {
	e.printStackTrace();
}

getInitialDriver("handtracking") will return the first driver that supports handtracking. If you would like to specify which driver you get data from, call getDevices("handtracking") and store it as a list, then call inflateDriver(devicePath, "handtracking"), where devicePath is one of the elements in the aforementioned list. Note that you must cast this driver as a HandTrackerInterface as shown above.

In order to use the EventManager, you will need to store an EventManager variable and write a receiver for the driver to hook onto. See Introduction to Receivers for the basic elements of receivers, and HandReceiver for the default class. To properly use a HandReceiver, you will need to create your own class to extend HandReceiver. For each function you would like to use, simply @Override that function, and then attach your receiver (in this case, a class named MyHandReceiver) to the driver, like so:

private EventManager eventManager;
...
eventManager = EventManager.getInstance();
receiver = new MyHandReceiver();
eventManager.registerReceiver(EventType.HAND_CREATED, receiver);
eventManager.registerReceiver(EventType.HAND_UPDATED, receiver);
eventManager.registerReceiver(EventType.HAND_DESTROYED, receiver);

This is an example HandReceiver class. This class stores a list of hands and their positions, and if a hand is destroyed, it removes the hand's data. Note that the receiver itself stores hand input, so getters need to be used.

public class MyHandReceiver extends HandReceiver {
	
	private Map<Integer, Coordinate3D> handPositions;
	
	public MyHandReceiver() {
		handPositions = new HashMap<Integer, Coordinate3D>();
	}
	
	public void handCreated(HandPosition handPos) {
		handPositions.put(handPos.id, handPos.position);
	}
	
	public void handUpdated(HandPosition handPos) {
		handPositions.put(handPos.id, handPos.position);
	}
	
	public void handDestroyed(int id) {
		handPositions.remove(id);
	}
	
	public Set<Integer> getHandIDs() {
		return handPositions.keySet();
	}
	
	public Coordinate3D getHandPosition(int id) {
		return handPositions.get(id);
	}
}

Finally, the last addition is that you must ensure that the driver updates and pushes events out. To do this, in your update or draw loop you should call #updateDriver on the handtracking driver instance.

Scaling Hand Data for Your Module

Unless your module only needs to interact with the top left corner of the screen, you'll want to scale your receiver's output to fit your screen. To do this, use the [HandTrackingUtilities] (https://github.com/ColoradoSchoolOfMines/interface_sdk/tree/master/src/main/java/edu/mines/acmX/exhibit/stdlib/input_processing/tracking/HandTrackingUtilities.java) functions getScaledHandX and getScaledHandY. These functions will trim a margin of the depth imaging window to increase sensitivity (and reduce the movement necessary to get across the screen), and scale its position to the specified resolution. Here is an example usage:

private int handX;
private int handY;
NOTE: width and height are accessed differently depending on the module type. These variables are keywords in Processing.
...
for (int id : receiver.getHandIDs()) {
	float marginFraction = (float) 1/6; //puts a 1/6 margin around the depth image (scales up 2/3 of depth image)
	handX = HandTrackingUtilities.getScaledHandX(receiver.getHandPosition(id).getX(), 
		handDriver.getHandTrackingWidth(), width, marginFraction);
	handY = HandTrackingUtilities.getScaledHandY(receiver.getHandPosition(id).getY(), 
		handDriver.getHandTrackingHeight(), height, marginFraction);
}

From here, you should can use the new hand data however you'd like. If you don't want a functionality (such as handDestroyed), simply omit it from your receiver. Only attach listeners for events you would like to receive. Have fun using hand tracking!