Modulant allows for the creation of images and their sonification.
The present implementation is built upon image-importing and freehand-drawing modules that may be used to create arbitrary visual scenes (and eerie sounds), with more constrained functional and typographical modules in development.
The audio engine is inspired by a 1940’s synthesizer, the ANS, that scans across images. In this scanning, one axis is time and the other axis is frequency. Modulant thus becomes a graphical space to be explored sonically and vice-versa. The project is built with Processing for graphics and interaction, and Ruby & Puredata for sound.
The early Modulant has a plain and minimalist interface, geared toward fast switching between its modes.
The easiest start for the new user is the toolbar, which facilitates selecting drawing modes and colours, playing and pausing the composition, and loading and saving the work.
The application also responds to keyboard shortcuts covering a richer set of functionality.
Here is the current list of keybindings:
Bindings for Drawing
r rectangle mode t triangle mode e ellipse mode f freehand mode p freehand paintbrush d freehand w/ dots b blur selected region Ctrl-z undo Ctrl-y redo Shift-l clear all
s selection mode Ctrl-c copy & interactively paste selection Ctrl-x cut & interactively paste selection Escape exit selection mode
Space resume/pause scanning Left/Right step 1px Ctrl-Left/Right step 10px Shift-Ctrl-Left/Right step 50px m mute/unmute sound g toggle grid
Escape-F1,F2,F3,F4 store current scanline position F1,F2,F3,F4 jump to stored position
Ctrl-o load background image Ctrl-s save work buffer as image
0 black (eraser) 1:8 Solarized base colours Ctrl-1:8 Solarized accents 9 white
h toggle help (this text) Shift-t toggle toolbar Ctrl-q exit application
Download and install Processing 3.0 (3.0a5 at the time of this writing):
Download and install (unpack) these libraries into your
See this short guide on installing libraries for the where and how if you are not sure.
The dependency on Processing 3.0 may go away at some point. Modulant gets a strange NullPointerException at startup on Processing 2.2.1.
Images that you load to sonify should have light features on dark backgrounds. The brightness of each pixel determines the volume of the associated oscillator -- if the majority of your image is bright, then all you're going to hear is a loud and harsh noise.
Also, when drawing, it's better to avoid bright and long/thick vertical lines/regions -- these will turn on a lot of oscillators in one go, which may sound unpleasant (unless you know what you're doing).
Blur is a nice tool to make sounds more gentle.
Example images for sonification
Included in the data directory:
- klee-lines-dots-drawing-bw.jpg : a drawing by Paul Klee (downloaded from here)
- sc02-1n.jpg : an image used with the actual ANS synthesizer (downloaded from here)
- Coil-ANS-C-grey.png : A "composition" by the experimental music group Coil
- Coil-ANS-D-grey.png : Another one by Coil (note that these are a bit noisy due to lack of image cleaning)
- graph2d-1.png : parametric graph 1
- graph2d-2.png : parametric graph 2
- graph2d-3.png : parametric graph 3
Modulant - A sonification and audiovisual performance interface experiment Copyright 2015 Berkan Eskikaya, Louis Pilfold This Source Code Form is subject to the terms of the Mozilla Public License, v. 2.0. If a copy of the MPL was not distributed with this file, You can obtain one at http://mozilla.org/MPL/2.0/.