"Sensory Fusion" is a concept in AI/robotics where inputs from multiple physical sensors are combined to reduce uncertainty and optimize the performance of machine perception. A typical application is in self-driving cars with real-time data from different sources such as LIDAR (Light Detection and Ranging), radar and infrared. Others are Internet-of-Things scenarios or the coordination of multiple robots. Usually stochastic methods such as Kalman filters or random sets are deployed.
In Sensory Con-Fusion, data streams from different modalities disturb and irritate each other, causing a synaesthetic nightmare. The project aims to challenge the predominant technological purpose of improvement and optimization. It invites debate about related topics like information overflow, autism and neural multisensory integration.
In this sketch, audio disturbs video: to see, be quiet.
Written in Processing 3 https://processing.org/. Libraries: P5 Audio and P5 Video.
This is still a work in progress (first version made: May 3, 2017). The focus is on the conceptual, not the aesthetic, and I have deliberately borrowed from the samples that come with Processing (see the sketch for details).
This was presented during "Multisensory Aesthetics", a symposium organized by Dr. Mike Blow at the University of Plymouth in May 2017. A presentation is attached.
I am still working on a VR version.
Copy this folder into your Processing sketch folder. You might have to install the P5 Video and P5 Audio libraries from the Processing GUI (Sketch -> Import Library -> Add Library).
Read the comments in the sketch files for details and current issues.
Processing is broken on Mac computers with Mac OSX 10.5 Catalina. For a workaround and to run this sketch with Processing 3.5.4 see: https://gist.github.com/i3games/b063987dfb62baf5d0afda422631b480
Processing 4.0 beta1 runs the sketch, get the latest version of Processing here https://processing.org/download