UI Handler Kinect

Table of Contents Black box description Bundles Features Design decisions Implementation Installation and setup instructions

Black box description

Pure gesture-based interaction using Kinect Kinect adapter supports gesture based interaction with the system. It receives messages via TCP socket and translates them to universAAL events.

Bundles

Artefact: UI Handler Kinect
Maven artefact	org.universAAL.ui / ui.kinect.adapter org.universAAL.ui / ui.kinect.adapter.instance
Pax Composite bundle	scan-composite:mvn:org.universAAL.ui/ui.kinect.adapter.instance/x.y.0/composite
Karaf Feature	-
Maven Site	https://universaal.github.io/ui/ui.kinect.adapter/index.html https://universaal.github.io/ui/ui.kinect.adapter.instance/index.html

Features

• ui.kinect.adapter acts as a library while ui.kinect.adapter.instance shows how to instantiate the adapter. It fills in the TCPServer socket where the adapter listens for the messages and decides how to connect the gestures with universAAL buses.

Design decisions

Following figure shows the overview of the adapter.

Green objects are part of the adapter. Red objects are universAAL specific. Blue objects are either .Net or general objects like sockets.

First a TCP connection is accepted by a TCP Server which gives the resulting Socket to a Worker for IO communication. In our case, UTF-8 encoded String Lines are transmitted over the Socket. The worker then forwards the message to a message parser, which parses the message into Objects, which are forwarded to a Broker. The Broker analyses the message like a network router, and forwards it to the right recipient which could be a ServiceCaller or ContextPublisher. In the end, the ServiceCaller or the ContextPublisher creates the universAAL specific objects from the received message and makes a call to the universAAL middleware.

Implementation

Current implementation forwards gestures defined in ont.handgestures ontology (from https://github.com/universAAL/ontology/wiki) to the context bus. Context Publisher publishes different Context Events so that another instance can subscribe to this events and do whatever upon receiving the event (e.g. to call some service etc.)

Class diagram of ui.kinect.adapter:

• The AbstractService and AbstractContext classes wrap the universAAL specific classes. Their purpose is to create the universAAL specific objects from the arguments received from the network.
• QueryService and RequestService just implement some functions of AbstractService to reduce code duplication.
• IServiceCall and IContextPublish are the interfaces for those classes which are ServiceCallers and Context publisher and make ServiceCalls or ContextPublishes using AbstractServices and AbstractContext objects.
• IContextCallback is an interface for the non universAAL context receivers, which receive objects instead of ContextEvents.
• The TCPServer, Worker and Message parser classes manage network communication and forward the received objects to an IMessageBroker implementation. The DefaultBroker implementation has a HashMap for finding the right AbstractService or AbstractContext.

Installation and setup instructions

1. Download programs needed for Kinect
2. Install programs in following order:

  * Open NI
  * Sensor Kinect Driver
  * Nite
  * Kinect SDK
  * Kinect Developer Toolkit

1. Maven install your components (be sure that ont.handgestures and ui.kinect.adapter are in your maven repository)
2. Run your (adapted?) ui.kinect.adapter.instance
3. Run KinectDtw (KinectDtw is the .Net framework for Kinect gesture recognition. It connects to the adapter via TCP.)

• Open the \Instances\KinectDtw\trunk\DTWGestureRecognition.sln in Visual Studio

or

• Run: \Instances\KinectDtw\trunk\DTWGestureRecognition\bin\Release\DTWGestureRecognition.exe

You may want to check the configuration in ui.kinect.instance (org.universAAL.kinect.adapter.osgi/Activator.java) and KinectDtw (Manager.cs)

By default ui.kinect.instance listens on port localhost:11111 and KinectDtw connects to it.

Be sure that:

• At all times only one coordinator is allowed.

org.universAAL.middleware.peer.is_coordinator=true

• For distributed setup on different machines the following lines should be commented like this:

1. cyberdomo.ssdp.mx=5
2. cyberdomo.ssdp.buffersize=2048
3. cyberdomo.ssdp.port=13531

• The following line identifies the space, so make sure to have it exactly the same in every runner

org.universAAL.middleware.peer.member_of=urn:org.universAAL.aal_space:test_environment

• Make sure UPNP is enabled
• Make sure firewall allows the connection.
• Check the IP/Port on TCP clients, and TCP Servers