Gesture Workflow
Start recording your composition (optional) by clicking the [o] (bang) in the top left section of the patch, next to area~. This should record to the project root, two binaural .wav files, containing the real and virtual audio capture.
More contextual information about musical parameter mappings can be found in the publication for this project. Moreover, an early visual guide to the gestures can be found in the explainer video provided, as well as the 360 binaural demonstration video with software overlay.
The recording or ‘sampling’ stage is initiated by making a left-hand grab above the LMC. The longer lasting the grab, the longer the portion of audio from the ambisonic palette is sampled. The three-dimensional coordinates of the hand above the LMC correlates with the location of audio recorded (this is achieved by mapping the hand coordinates to a virtual microphone inside the ambisonic palette), essentially allowing the user to record sounds around their person in three dimensions. Upon letting go of the grab gesture, the sample plays on repeat (using the karma~ Library) through the bone conduction headphones, thus setting up the session’s virtual audio environment.
The manipulation stage is automatically initiated after the ending of the previous grab gesture and uses translational (x, y, z) and rotational (roll, pitch) values from both hands when above the LMC. There are two audio effects being manipulated, with parameters from these effects mapped in different ways to the translation and rotation of the user’s hands.
The spatialise stage begins once the manipulation stage is ended by the user. The three-dimensional space above the LMC is mapped to the virtual audio environment, in which the user is currently listening to the sample that they have recorded. The user can use their right hand to move the sample around the virtual audio environment. For an example of the effect this has, moving the hand between the two extremes of the x-axis (left to right) results in hearing the sample move from ear to ear. The spatialise stage is ended by grabbing with the right hand.
To summarise, the patch can be categorised into having two inputs: audio from the user’s environment and hand gesture, and one output: the virtual audio environment. In the background, this audio input is decoded into the ambisonic palette (inaudible), which is acted on by the user’s hands to form one audible output: the virtual audio environment, which is comprised of up to 8 nodes. Through the choice of sensory overlay (bone conduction) and integration of head tracking, this virtual audio environment is experienced synchronously with the user’s real, multisensory environment.