Repository for the sensory subtitution project
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Type Name Latest commit message Commit time
Failed to load latest commit information.

Open Sensory Substitution by Interspecifics

The term sensory substitution refers to a non-invasive technique introduced by neuroscientist Paul Bach-y-Rita in the 60's who attends restoring senses damaged using hybrid interfaces brain-machine to redirect certain characteristics of a sensory modality to another and using the adaptive capacities of the brain to replace perceptual categories. He suggests that while the brain and its connection with the environment operate properly, peripheral sensory systems are secondary in the exercise of global cognition.

the system

The aim of the prototype of sensory substitution is to allow the brain to create a mapping of space through sound and electrotáctiles sensations. And thus consider expanding human capabilities and challenge the Aristotelian conception that postulates five senses with specific functions clearly separated from each other.

In this project we are using OpenCV to detect blobs from a video camara attached to the Raspberry Pi, this detections is later tranform into sound using SuperCollider and SC.VO.3.1 for python and electrical impulses using GPIOS.

The kit The kit


  • RaspberryPi B+
  • Camara module for raspberry
  • Micro sd cards
  • Electro-tactil transducers
  • Lithium baterrys for the raspberry


The following image contains all the libraries necessary to round the OSS, incluiding the python scripts to comunicate OpenCV with SuperCollider.

#Libraries and installs

  • OpenCV for python OpenCV (Open Source Computer Vision Library) is an open source computer vision and machine learning software library. OpenCV was built to provide a common infrastructure for computer vision applications and to accelerate the use of machine perception in the commercial products. Being a BSD-licensed product, OpenCV makes it easy for businesses to utilize and modify the code.

  • SuperCollider: SuperCollider is a programming language for real time audio synthesis and algorithmic composition. The language interpreter runs in a cross platform IDE (OS X/Linux/Windows) and communicates via Open Sound Control with one or more synthesis servers. The SuperCollider synthesis server runs in a separate process or even on a separate machine so it is ideal for realtime networked music.

  • SC.V0.3.1 The scosc module converts standard python types to supercollider osc messages. In order to play sounds in musical time, we have to send OSC messages with timestamps. Because the server has a limited message buffer (1024), we have to send messages as a stream in real-time. The challenge is getting the messages in early enough that the server can process them, while not sending too many as to override the buffer. We also have to take latency into account, which makes this a much bigger bundle of fun.


  • $ ssh pi@sensory1.local
  • $ password: 1234
  • $ cd B10s_Senses
  • $ cd B10s_Senses/Python
  • $ cd B10s_Senses/Python/examples
  • $ cd ..
  • $ cd B10s_Senses/Python
  • $ ./
  • $ sudo python

#Python + SuperCollider examples

In the following example with show a regular synthdef from supercollider using the sclang lenguage:

SynthDef("PMCrotale", {
arg midi = 60, tone = 3, art = 1, amp = 0.8, pan = 0;
var env, out, mod, freq;
freq = midi.midicps;
env = Env.perc(0, art);
mod = 5 + (1/IRand(2, 6));
out =, mod*freq,
   pmindex:, timeScale: art, levelScale: tone),
   mul:, timeScale: art, levelScale: 0.3));
out =, pan);
out = out *, timeScale: 1.3*art,
   levelScale: Rand(0.1, 0.5), doneAction:2);, out); //, out);

Synth("PMCrotale", ["midi", rrand(48, 72).round(1), "tone", rrand(1, 6)])

In this case we invoque using a regular synth argument.

Using python tu interface:

import sc
import time, random

seed = random.Random()

sc.start( verbose=1, spew=1 )
s = ###################

PMCrotale = sc.Synth( "PMCrotale" )
PMCrotale.freq = 400

timeout = time.time() + 10 # 10 secs

while time.time() < timeout :
   time.sleep(0.5) # loop every half sec
   PMCrotale.midi = seed.randint( 40, 90 )
   PMCrotale.tono =  10 = 1

print 'seeya world! ...........'


The following maps were designed during the workshop of sensory substitution in the UDK Berlin. The idea to expand the possibilities of the system in search of a systematic process that also allows sensory expansion.



  • Paloma López
  • Leslie García
  • Thiago Hersan