GestureKit

A Swift Package to detect Hand Gestures.

Table of contents

1. Introduction
2. How it works
3. Benefits
4. Installation
5. Example Code
   1. Gesture Detection Example Code
   2. Virtual Hands Example Code
6. Examples
7. Info.plist
8. Appendix
9. About me

Introduction

Gestures can be composed and exported from the VisionOS Gesture Composer App

How it works

The GestureComposer App exports *.gesturecomposer packages, we can pass these to the GestureDetector which will detect if the user has performed a particular gesture.

Benefits of using GestureKit

• You don't need to manually program gestures
• It's faster to create and test gestures
• Gestures can be easily shared between Apps
• Hand gestures can have multiple steps
• Multiple gestures can be detected at a time, for instance, one's left hand could perform a gesture different to one's right hand
• GestureComposer packages contain *.USDZ Animated 3D files of the composed gesture to optionally show users what to do with their hands
• The file-format is straight forward, ready for expansion ✨

Installation

GestureKit is best installed via Xcode's Package Manager with the following URL

https://github.com/nthState/GestureKit

Note If you install via copying the project directly, you may get a UUIDV7 error, you will also need to include the following package

https://github.com/nthState/UUIDV7

Example Code

Gesture Detection Example Code

This example code will create a RealityKit View and run a task to detect the gesture packages set in the configuration.

Note: This must be ran in an Immersive Space on VisionOS.

import GestureKit
import RealityKit
import SwiftUI

struct ContentView {

  let configuration = GestureDetectorConfiguration(packages: [
    URL(fileURLWithPath: "/path/to/some/package.gesturecomposer"),
    URL(fileURLWithPath: "/path/to/some/other.gesturecomposer")
  ])
  
  let detector: GestureDetector
  
  init() {
    detector = GestureDetector(configuration: configuration)
  }

}

extension ContentView: View {

  var body: some View {
    realityView
  }

  private var realityView: some View {
    RealityView { _ in

    }
    .task {
      for await qesture in detector.detectedGestures {
        print("Gesture Detected: \(qesture.description)")
      }
    }
  }

}

Virtual Hands Example Code

Part of development required to show virtual hands for debugging, I found this useful so it is bundled as part of GestureKit.

Note: This must be ran in an Immersive Space on VisionOS.

import GestureKit
import RealityKit
import SwiftUI

struct ContentView {

  let virtualHands: VirtualHands
  
  init() {
    let handsConfiguration = VirtualHandsConfiguration(left: HandConfiguration(color: .blue, usdz: HandConfiguration.defaultModel(chirality: .left)), 
    right: HandConfiguration(color: .red, usdz: HandConfiguration.defaultModel(chirality: .right)),
    handRenderOptions: [.model, .joints, .bones])
    virtualHands = VirtualHands(configuration: handsConfiguration)
  }

}

extension ContentView: View {

  var body: some View {
    realityView
  }

  private var realityView: some View {
    RealityView { content in
    
      let root = Entity()
      content.add(root)
    
      do {
        let (left, right) = try virtualHands.createVirtualHands()
        
        rootEntity.addChild(left)
        rootEntity.addChild(right)
      } catch {
        print("Failed to add virtual hands to simulation: \(error.localizedDescription)")
      }

    }
    .task {
      await virtualHands.startSession()
    }
    .task {
      await virtualHands.startHandTracking()
    }
    .task {
      await virtualHands.handleSessionEvents()
    }
  }

}

Examples

Gesture	Asset
On your Right hand, Move your finger tips together, then move your ring and middle finger to your plam	Download the Spiderman example
On your Right hand, Move your finger tips together, then put your middle finger tip and ring finger tip together, then separate them	Download the Vulcan Salute example
On your Right hand, Touch your ring finger tip and thumb tip on together	Download the ring-thumb-tip-touch example

Info.plist

You will also need to add the following keys to your Info.plist

<key>NSWorldSensingUsageDescription</key>
<string>Insert your description here</string>
<key>NSHandsTrackingUsageDescription</key>
<string>Insert your description here</string>

Appendix

• Download Gesture Composer from the VisionOS AppStore
• Browse & Download Gestures from the Gesture Composer Website
• Follow me on Twitter/X

About me

Hello, I'm Chris, I created Gesture Composer, it has the following components:

• Website
• APIs
• VisionOS App
• Swift Package

It's been fun to make, I hope it's useful to you.