Sanchitgarg/audio sensor #1646

sanchitgarg · 2022-02-02T00:52:51Z

Motivation and Context

With this change, we give our awesome virtual agents the ability to hear. This enables new research avenues like Audio-Visual navigation.

We are adding a new git-submodule : rlr-audio-propagation.
This git repo hosts the header and binaries for Meta RL Research's audio propagation engine. A new AudioSensor C++ class is added that calls into this new lib. The C++ AudioSensor class is exposed to python via the existing pybindings.

Further, we are adding a new build flag : --audio.
This is an optional flag. Any user who wants to use the audio sensor will have to build with this flag.

Please see the examples/tutorials/audio_agent.py script for a tutorial on how to use it.
Also, please refer to docs/AUDIO.md which goes into more details about the feature.

How Has This Been Tested

The code was tested locally using the C++ viewer. A new audio_agent.py script has been added as well to test the change. Further, we tested the integration with the SoundSpaces team. They were able to generate impulse response in they navigation system.

Types of changes

Docs change / refactoring / dependency upgrade
Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to change)

Checklist

My code follows the code style of this project.
My change requires a change to the documentation.
I have updated the documentation accordingly.
I have read the CONTRIBUTING document.
I have completed my CLA (see CONTRIBUTING)
I have added tests to cover my changes.
All new and existing tests passed.

src/esp/bindings/CMakeLists.txt

aclegg3

Thanks for creating this PR, @sanchitgarg. Just a quick pass with some easy clean-up. Will return with a more thorough review.

data/test_assets/dataset_tests/dataset_0/objects/dataset_test_object1.object_config.json

src/cmake/dependencies.cmake

src/esp/bindings/SensorBindings.cpp

examples/tutorials/audio_agent.py

src/esp/bindings/SensorBindings.cpp

src/esp/core/CMakeLists.txt

src_python/habitat_sim/simulator.py

src/esp/assets/ResourceManager.cpp

.gitmodules

aclegg3

Did a proof pass over the README. Thanks for the documentation, this is great!

docs/AUDIO.md

aclegg3 · 2022-03-30T17:51:32Z

docs/AUDIO.md

+
+The C++ Audio sensor class uses the [RLRAudioPropagation](https://github.com/facebookresearch/rlr-audio-propagation) library. This is a bi-directional ray tracer based audio simulator. Given a source location, listener location, mesh, audio materials and some parameters will simulate how audio waves travel from the source to the listener. The output of the library is an impulse response for the given setup. 
+
+The C++ implementation is exposed for python user via py-bindings. This document explains the various python apis, structs and enums.


Note: once this is live on aihabitat.org, we can point directly to the AudioSensor class docs, tutorial Colab, or other resources too.

Suggested change

The C++ implementation is exposed for python user via py-bindings. This document explains the various python apis, structs and enums.

The C++ implementation is exposed for python users via pybind11. This document explains the various python APIs, structs, and enums. Also see the relevant [Habitat-sim python API](https://aihabitat.org/docs/habitat-sim/classes.html) doc pages.

docs/AUDIO.md

aclegg3

Looks good overall.

A few documentation comments and I didn't test the viewer changes (CI won't either), so please double check viewer with and without the audio build.

Once those are resolved and the CI is green I think we can merge. 🎉

src/esp/bindings/SensorBindings.cpp

src/esp/sensor/AudioSensor.h

src_python/habitat_sim/simulator.py

Co-authored-by: Aaron Gokaslan <aaronGokaslan@gmail.com>

aclegg3

When tests pass, good to merge!

facebook-github-bot added the CLA Signed Do not delete this pull request or issue due to inactivity. label Feb 2, 2022

Skylion007 reviewed Feb 2, 2022

View reviewed changes

src/esp/bindings/CMakeLists.txt Show resolved Hide resolved

aclegg3 reviewed Feb 2, 2022

View reviewed changes

data/test_assets/dataset_tests/dataset_0/objects/dataset_test_object1.object_config.json Outdated Show resolved Hide resolved

src/cmake/dependencies.cmake Outdated Show resolved Hide resolved

Skylion007 reviewed Feb 2, 2022

View reviewed changes

src/esp/bindings/SensorBindings.cpp Outdated Show resolved Hide resolved

sanchitgarg force-pushed the sanchitgarg/audio-sensor branch from 7368216 to 7067b0a Compare February 2, 2022 21:13