Skip to content
This repository

A simple, yet configurable Kinect-based toolkit for control of musical instruments

branch: master

Fetching latest commit…

Octocat-spinner-32-eaf2f5

Cannot retrieve the latest commit at this time

Octocat-spinner-32 Kinesthesia
Octocat-spinner-32 KinesthesiaModelingProject
Octocat-spinner-32 .gitignore
Octocat-spinner-32 Kinesthesia.sln
Octocat-spinner-32 README.markdown
Octocat-spinner-32 readme.html
Octocat-spinner-32 readme.rtf
README.markdown

Kinesthesia

a simple yet configurable toolkit that works with Kinect and interprets gesture recognition data into MIDI signals


video demos


Versions

v 0.5 (current version)

  • Gesture recognition is now handled by KinectToolbox, which gives faster response and more customizable properties for gestures.
  • Configuration interface for gesture behavior. (see Configuration Files part for more info)
  • Voice command control (see Voice Command Controls part for more info)
  • Save / load / refresh / restore / quick save / quick load methods for configuration files (see Configuration Files part for more info)

v 0.4

  • MIDI-player which supports MIDI-files parsed to CSV (see Parsing MIDI to CSV part for more info)
  • Multiple MIDI-tracks control support
  • Velocity change control for each hand (e.g. left hand changes the velocity for the piano part played by left hand and so on)
  • File browser and simple CSV-parser

v 0.3

  • Simplified gesture recognition
  • Additions in MIDI-class

v 0.2

  • Kinect skeletal tracking basics
  • Interpreting hands coordinates as notes

v 0.1

  • Basic MIDI-interaction
  • Some base classes

Configuration Files

Gestures and properties

Kinesthesia provides a simple configuration interface for better control on gesture's behavior. Currently there're four types of gestures binded for each hand, they're:

  • SwipeUp
  • SwipeDown
  • SwipeToRight
  • SwipeToLeft

Each type is handled by leftHandSwipeDetector and rightHandSwipeDetector which are binded to left and right hands correspondingly. In order to configure detector's behavior, there're 6 properties:

  • Swipe Minimal Length
  • Swipe Maximal Length
  • Swipe Minimal Height
  • Swipe Maximal Height
  • Swipe Minimal Duration
  • Swipe Maximal Duration

where:

  • Length - minimal or maximal horizontal path length for swipe gesture
  • Height - minimal or maximal vertical path length for swipe gesture
  • Duration - minimal or maximal duration for swipe gesture

These properties can be configured with appropriate sliders for each hand. Length and Height are doubles and their range is from 0 to 1 (because original coordinates provided by Kinect SDK are very small floating point numbers). Duration is measured in miliseconds and its range is from 0 to 3000.

Generated configuration Files

In order to load default configuration, Kinesthesia loads default.csv from the current directory where executable file is located.

Kinesthesia also creates a temporary backup file temp.csv, which is rewriten every time when new configuration file is loaded.

When quicksaving, Kinesthesia creates a new .csv configuration file using current UNIX time.

When closing, Kinesthesia creates or overwrites cache.txt file, which contains paths for the latest used configuration / MIDI files.

Custom configuration files

Configuration file can be created in any text editor or even edited withing the Settings Log block within the software. It should have .csv extension and follow this standard:

The first two lines are configuration lines for two gesture detectors:

HandRight, 0.4, 0.2, 0.1, 0.2, 250, 1500
HandLeft, 0.4, 0.2, 0.1, 0.2, 250, 1500

where the parameters are:

  • Hand that will be tracked: HandRight or HandLeft
  • Swipe Minimal Length
  • Swipe Maximal Length
  • Swipe Minimal Height
  • Swipe Maximal Height
  • Swipe Minimal Duration
  • Swipe Maximal Duration

The remaining 8 lines cover events and methods which should be called.

HandRight, SwipeToRight, SendNote
HandRight, SwipeToLeft, SendNote
HandRight, SwipeUp, SendNote
HandRight, SwipeDown, SendNote
HandLeft, SwipeToRight, ChangeVolume
HandLeft, SwipeToLeft, ChangeVolume
HandLeft, SwipeUp, BendPitch
HandLeft, SwipeDown, BendPitch

The parameters are:

  • Hand that will be tracked: HandRight or HandLeft
  • Type of Swipe Gesture: SwipeUp, SwipeDown, SwipeToLeft, SwipeToRight
  • Method that will be called when this gesture will be detected: SendNote, BendPitch, ChangeVolume, VelocityChange (available only for MIDI tracks) and DoNothing (if you don't want to call any method)

Quick Loading / Quick Saving / Refreshing / Restoring defaults

  • Quick Loading -- loads the last configuration or MIDI file. The path is stored in cache.txt
  • Quick Saving -- saves current configuration to the current folder by creating a .csv file with current unix time as file's name
  • Refreshing -- reloads Settings log, so it could match the changes in detector's properties
  • Restoring defaults -- reloads configuration from default.csv file

Voice Command Controls

Currently, this project uses voice recognition class provided by KinectToolbox. Kinesthesia can recognize the following commands:

  • Play -- plays the current parsed MIDI track
  • Stop -- stops playing current track
  • Swipe -- turns on gesture recognition (unfortunately the library wasn't able to recognize the word "track")
  • Load -- load last configuration file
  • Save -- quickly save current configuration file
  • File -- load last MIDI(CSV) file
  • Reload -- reload settings log
  • Default -- restore default configuration

Parsing MIDI to CSV

To parse a binary MIDI file to CSV, you'll need this command line tool

Just copy the MIDI file to the folder with Midicsv.exe and execute command like this:

Midicsv.exe example.mid example.csv

where the first item is the converter itself, second is the MIDI-file and third is the desired output name

Then, choose the resulted MIDI-file via Open dialogue. Currently this file should have at least three tracks, otherwise the program woud crash. However you can change the quantity of tracks within the MainWindow.xaml.cs class


for Kinect and WPF-related stuff I use:

  • KinectSDK
  • KinectToolbox
  • Examples from KinectforWindowsSDK series

for MIDI-related stuff I use this awesome library: midi-dot-net

for virtual port I'm currently using loopbe1

however there's also a good analogue for it: midiyoke

Something went wrong with that request. Please try again.