Skip to content

IO Framework Integration for hand tracking devices

Jump to bottom
rahul edited this page Nov 1, 2016 · 9 revisions

Integration for IO Devices with hand tracking capabilities

Pre-requisites:

  • A stable version of Android Studio. The library and the sample app have been tested on version 2.1.3.
  • For devices with a native library the template device has been tested with ndk version r10e.

Instructions:

Setting up the Application

  • Download the following "aar" files from here and rename them as below:

    • 3DCursorLibrary-debug-XX-XX-XXXX.aar -> 3DCursorLibrary-debug.aar
    • backend_oculus-releaseToGitHub-XX-XX-XXXX.aar -> backend_debug.aar
    • framework-releaseToGitHub-XX-XX-XXXX.aar -> framework-debug.aar
    • GearWearIoDevice-debug-XX-XX-XXXX.aar -> GearWearIoDevice-debug.aar
    • gearwearlibrary-debug-XX-XX-XXXX.aar -> gearwearlibrary-debug-.aar

  • Download the GVRf repository along with the demo apps (links below):

  • Copy the downloaded aar files to the following location: GearVRf-Demos/gearvrf-libs/

  • Open Android Studio and select the gvr-3dcursor-simple project from the demo app repository 3d cursor simple sample

  • Download the Oculus Mobile SDK 1.0.3 and copy the necessary files to the application (create the necessary missing folders):

ovr_sdk_mobile/VrApi/Libs/Android/VrApi.jar to /app/libs/VrApi.jar
ovr_sdk_mobile/VrAppSupport/SystemUtils/Libs/Android/SystemUtils.jar to app/libs/SystemUtils.jar
ovr_sdk_mobile/VrApi/Libs/Android/armeabi-v7a/libvrapi.so to app/src/main/jniLibs/armeabi-v7a/libvrapi.so
```

### Integration
To make it easy to integrate we have placed tags with _VENDOR_TODO_ throughout the code (in both the framework and the sample app) for lines that needs changing.  You can easily locate all such files using:

`$ grep -rnw "_VENDOR_TODO_" .`

Now, let’s assume that we need to add the following device to 3D Cursor:

Company Name: XYZ Company VendorId: 1234 ProductId: 5678


1. First make a copy of the io_hand_template folder from the GVRf Extensions folder:
`GearVRf/GVRf/Extensions/3DCursor/IODevices/io_hand_template -> GearVRf/GVRf/Extensions/3DCursor/IODevices/xyz_company`

2. In the folder xyz_company, modify the vendor_info.txt file to match your device. This readable text file makes it easy for app developers to add this IO device to their settings.xml.

  1. name="Left Hand" deviceId="left_INDEX" vendorId="1234" productId="5678" vendorName="XYZ Company"

  2. name="Right Hand" deviceId="right_INDEX" vendorId="1234" productId="5678" vendorName="XYZ Company"

3. Next, add the following line to the end of the application's settings.gradle file

include ':IODevices:xyz_company' project(':IODevices:xyz_company').projectDir = new File("../../GearVRf/GVRf/Extensions/3DCursor/" +         "IODevices/xyz_company")


Make sure that the path to the IO device module is correct. Next, hit “Sync Now” on Android Studio. The new folder should appear as an Android Studio module.

4. Now rename the TemplateHandDevice.java class (which can be found in the newly added module "xyz_company") to XYZHandDevice.java

5. Change the java class to match the information provided in the vendor_info.txt file (updated in step 2):

private static final int VENDOR_ID = 1234; private static final int PRODUCT_ID = 5678; private static final String VENDOR_NAME = "XYZ Company";


6. The template device feeds mock position values from the native layer to the hand model. Update the HandTemplateDevice class to translate input events from your input sensor to values for the hand model created by the template (look at the processData() method). Next, in order to move the cursor make sure to call the setPosition(float x, float y, float z) call in the class to feed x, y and z values to the cursor (here we manipulate the cursor using the position values from the TIP joint of the INDEX finger).

You can assume a fixed coordinate system while generating the x, y and z values. 

* Button presses can be fed using setKeyEvent(KeyEvent keyEvent). 
* Rotation values can be provided using the setRotation(float w, float x, float y, float z).
* For convenience we have included a native file (template_hand_device.cpp) to generate hand tracking events for the Java classes. Use it if required.

7. When the device has been initialized make use of the setConnected(true) method to let the framework know that the device is ready. This is followed by setConnected(false) on de-initialization. The corresponding calls can be made in the HandIODevice class.

8. Next, add the following line to build.gradle file for the app:

dependencies { compile fileTree(include: ['*.jar'], dir: 'libs') compile(name:'3DCursorLibrary-debug', ext:'aar') compile project(':IODevices:xyz') }

Again hit sync now on Android Studio.

9. Now in the app module add the following lines inside onInit() method of CursorMain.java. Note that the second argument for the constructor is the deviceId:
 ````
 XYZHandDevice device = new XYZHandDevice(gvrContext, mainScene);
 devices.addAll(device.getDeviceList());
 ````
10. Finally in the settings.xml file(GearVRf-Demos/gvr-3dcursor-simple/app/src/main/assets/settings.xml) modify the <io> tag to use the new device:

 ````
 <cursor name="Left Cursor"
         active="yes"
         position="center"
         themeId="crystal_sphere"
         type="object">
     <io deviceId="left_INDEX"
         priority="1"
         productId="5678"
         vendorId="1234"/>
 </cursor>
 <cursor name="Right Cursor"
         active="yes"
         position="center"
         themeId="crystal_sphere"
         type="object">
     <io deviceId="right_INDEX"
         priority="1"
         productId="5678"
         vendorId="1234"/>
 </cursor>
 ````
 Additionally, make sure that the active tag on any remaining cursors is “no” if you do not want them to show up in your application.

 ````
 <cursor name="Left Cursor"
         active="no"
         position="center"
         themeId="crystal_sphere"
         type="object">
 ````

## Post Integration Steps (Optional):

1. Rename the ndk module name and the cpp file to match the newly added device:

* template_hand_device.cpp -> xyz_device.cpp
* In TemplateHandDevice.java:
   ````
   static {
   System.loadLibrary("xyz_device");
   }
   ````
* In the module’s build.gradle:
   `moduleName = "xyz_device"`

2. Change the package name for the newly created module:
`com.sample.template -> com.sample.xyz`

3. Add a close call to CursorMain.java inside close():

device.close();


4. To test the app, add the oculus signature of the phone to asset folder. The oculus signature can be obtained here:
a. https://developer.oculus.com/osig/

5. The same integration can be tested using the [3d cursor sample](https://github.com/gearvrf/GearVRf-Demos/tree/master/gvr-3dcursor) as well.
Clone this wiki locally