[ASR] Fix for multi-channel signals in AudioSegment

[anteju]

Signed-off-by: Ante Jukić ajukic@nvidia.com

What does this PR do ?

Currently, AudioSegment aims to do channel averaging for multi-channel audio signals:

if self._samples.ndim >= 2:
  self._samples = np.mean(self._samples, 1)

However, samples in its constructor are arranged as (num_channels, num_samples), instead of the assumed (num_samples, num_channels). This means that samples of the created AudioSegment will have an unexpected size, and also ASR will fail when running transcribe on a multi-channel audio signal.

This can be tested by running test_audiosegment.py and test_asr_multi_channel_audio.py from the attached scripts in test_audiosegment_with_channel_selector.zip

Collection: ASR

Changelog

Necessary fix in AudioSegment

• correction to dimension ordering for multi-channel audio
• added a new parameter channel_selector which can be used to select a single-channel (or a subset of channels) from a multi-channel audio signal
• added unit tests for both channel selector and AudioSegment

Changes above are in segment.py and audio_utils.py and tests are in test_preprocessing_segment.py and test_audio_utils.py.

Nice to have: channel_selector for transcribe

• Added a channel_selector option to transcribe, and forwarded it through models, audio-to-text dataset and WaveformFeaturizer to AudioSegment
• This is a convenience when running an existing model on a multi-channel audio file, we can select either one of the channels or the channel-wise average.

If preferred, this can be removed from the PR.

Usage

• Check the attached scripts: test_audiosegment_with_channel_selector.zip

Example: AudioSegment

• Check test_audiosegment.py

AudioSegment.from_file(audio_file) # loads the complete signal (including all channels)
AudioSegment.from_file(audio_file, channel_selector='average') # loads a single-channel signal (average across channels)
AudioSegment.from_file(audio_file, channel_selector=0) # loads only first channel

Example: AudioSegment

• Check test_asr_with_channel_selector.py

asr_model.transcribe(mc_files, channel_selector='average') # apply channel averaging when loading audio

Tests

• Existing unit tests + newly-added unit tests
• Compared transcription using single-channel signal matches between main and fix branch for wav, flac, mp3 input (test_asr_formats.py, attached)
• Compared transcription of a channel from a multi-channel signal is matching transcription using a single-channel input signal (test_asr_with_channel_selector.py, attached)
• Tutorials ASR_with_NeMo.ipynb and ASR_with_Transducers.ipynb running fine with the fix branch

Before your PR is "Ready for review"

Pre checks:

• Make sure you read and followed Contributor guidelines
• Did you write any new necessary tests?
• Did you add or update any necessary documentation?
• Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  • Reviewer: Does the PR have correct import guards for all optional libraries?

PR Type:

• New Feature
• Bugfix
• Documentation

If you haven't finished some of the above items you can still open "Draft" PR.

Who can review?

Anyone in the community is free to review the PR once the checks have passed.
Contributor guidelines contains specific people who can review PRs to various areas.

Additional Information

• n/a

[lgtm-com]

This pull request introduces 1 alert when merging 809c71a into d969162 - view on LGTM.com

new alerts:

• 1 for Unused import

[jbalam-nv]

Looks great