This repository contains sound synthesis code used by our papers:
Shape and Material from Sound, NIPS 2017
Project page: http://sound.csail.mit.edu/
Our code has been tested on 64-bit Ubuntu 14.04.5.
We provide auto install scripts for all required libraries except Matlab & MKL. You need to modify
auto_install.sh to indicate the paths to MKL libraries, libiomp5 (often included in an MKL install), and Matlab.
Then, simply run
./auto_install.sh (requires sudo privileges). This should install all the necessary components for you to synthesize sound, including
If you prefer to install the dependencies manually (e.g., you don't have sudo access), please refer to the manual installation section at the end of the document for instructions on customized installation and usage.
After auto installation, run (with Python 3 and FFmpeg)
python online_synth/gen_sound.py -r 1000 101 4 1 2
You will find the result in
result/scene-1000/obj-101-1/mat-4-2. The generated audio should sound like this. Here,
-r indicates rendering audio only (no video). The first number
1000 is the ID of the scene. Each of the following pair describes an object (its ID and material):
101 is for the floor, and
4 indicates it's made of wood;
1 is for the object (plate) and
2 is for its material (steel). You can also add in more objects by specifying different parameters.
To render the corresponding video, you need to specify the path to Blender in
gen_sound.py. Then, run the same command but this time without
python online_synth/gen_sound.py 1000 101 4 1 2
It should now render the video as well. The generated video should look like this.
To play with more objects & materials, run
get_precompute_data.sh to download our pre-computed data. All possible object/material combinations are listed in
Object data after precomputation (6.3G): all possible object and material combinations are in
data/obj_mat_list.txt. Please place the unzipped folder in
get_precompute_data.sh will download the pre-compute data and place it in the right place for you.
Sound-20k (Soundtrack only, 25G): the dataset used in our ICCV 2017 paper. Metadata can be found in
The code is structured as two separate parts: offline pre-computation and online synthesis.
The definitions of scenes and materials are located in the
Our pre-computation step uses two different solvers for calculating sound propagation: a direct BEM solver and an accelerated BEM FMM solver. In short, the direct method is faster when your mesh has a small number of faces, and the FMM solver is faster and more memory-efficient when the mesh has more faces. We suggest using the direct solver for meshes with less than 1000 faces.
One can run pre-computation using the direct method via
pre_compute/run_precalc_bem.sh or using the FMM solver via
pre_compute/run_precalc.sh. Note that you need to pass an object id followed by a material id to the script. The structure of the data folder should be
data/DATASET_NAME/OBJECT_ID; all generated files would be in
data/DATASET_NAME/OBJECT_ID/MATERIAL_ID. For example
./run_precalc.sh 0 1
will pre-compute object 0 with material 1 defined in
config/material/material-1.cfg using the FMM solver.
Running online simulation
The main file is
As described above, for video rendering, you need to install Blender and specify its path in
gen_sound.py. Also FFmpeg is used to combine separate sound tracks or rendered images into a single soundtrack / video.
gen_sound.py has some options. For example, one can use
-r to skip rendering and
-v to skip combining rendered images and sound tracks into a single video. Please refer to the help function for a detailed explanation.
To use the script, one needs to specify the scene id, and a number of paired object id and material id, as
gen_sound.py scene-id object1-id object1-material-id object2-id object2-material-id
Note: This section is only for users who choose not to use our auto-installation script.
Building Modal Sound
One can run
modelsound/auto_install.sh to install all required libraries and build all the necessary binaries.
If you wish to manually install the libraries or to customize their locations, please see this document for a detailed description.
Building file generators
Installing BEM/FMM Solver
We solve the Helmholtz equation related to sound propagation using a direct BEM method and an accelerated version using FMM.
- For the direct BEM method, one needs to install the Nihu Matlab library and specify its path in
- For the FMM BEM method, one needs to install the FMMlib3d libraries and specify its path in
An auto-install script is provided in
pre_compute/external/auto_install.sh. Note that you need to specify your Matlab installation path in the first line of
- enter gmake, run
- The built binary is located in
If you need to modify our simulation code, they are located in