Torchaudio resampling could be faster and simpler

[adefossez]

🐛 Bug

I have been implementing a few DSP algorithms lately, including sinc based resampling.
The following implementation could simplify the code in torchaudio and is also faster (single call to conv1d, instead of multiple calls to conv1d and conv_transpose1d). I'll be happy to open a pull requests if you are interested.

https://github.com/adefossez/julius/blob/main/julius/resample.py

To Reproduce

Steps to reproduce the behavior:

1. Install modules pip install -U torch torchaudio julius
2. Run the benchmark code hereafter

from torchaudio.compliance import kaldi
import torch
import julius
x = th.randn(1, 44100 * 10)
rolloff = 0.99  # lowpass filter freq used by torchaudio
zeros = 6  # use the same number of zero crossing as torchaudio
for from_sr, to_sr in [(5, 7), (7, 5)]:
    print("comparing for", from_sr, to_sr)
    %timeit kaldi.resample_waveform(x, from_sr, to_sr)
    %timeit julius.resample_frac(x, from_sr, to_sr, rolloff=rolloff, zeros=zeros)
    yt = kaldi.resample_waveform(x, from_sr, to_sr)
    yj = julius.resample_frac(x, from_sr, to_sr, rolloff=rolloff, zeros=zeros)
    print(torch.norm(yt - yj))

Expected behavior

Would expect torchaudio to be as fast as julius, results are

comparing for 5 7
32.4 ms ± 309 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
4.46 ms ± 44.9 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
tensor(8.7085e-05)
comparing for 7 5
12.9 ms ± 323 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
2.83 ms ± 43.9 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
tensor(5.1333e-05)

Environment

• What commands did you used to install torchaudio (conda/pip/build from source)? pip
• If you are building from source, which commit is it?
• What does torchaudio.__version__ print? 0.7.0a0+ac17b64

PyTorch version: 1.7.0
Is debug build: True
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: macOS 10.15.7 (x86_64)
GCC version: Could not collect
Clang version: 11.0.0
CMake version: version 3.18.4

Python version: 3.8 (64-bit runtime)
Is CUDA available: False
CUDA runtime version: No CUDA
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.19.2
[conda] blas 1.0 mkl
[conda] mkl 2019.4 233
[conda] mkl-service 2.3.0 py38hfbe908c_0
[conda] mkl_fft 1.2.0 py38hc64f4ea_0
[conda] mkl_random 1.1.1 py38h959d312_0
[conda] numpy 1.19.1 py38h3b9f5b6_0
[conda] numpy-base 1.19.1 py38hcfb5961_0
[conda] pytorch 1.7.0 py3.8_0 pytorch
[conda] torchaudio 0.7.0 py38 pytorch

[adefossez]

See adefossez/julius#4 for context.

[vincentqb]

That's great :) Yes, faster resampling is of interest to us, e.g. #908 and #911. As mentioned in comment by @mthrok, it may be appropriate in the long term to consider a C implementation instead of python though.

[vincentqb]

In terms of testing, you can look at the testing readme, but tests should catch support for batching and torchscript among a few things.

Grepping shows:

batch_consistency_test.py:    def test_batch_Resample(self):
batch_consistency_test.py:        expected = torchaudio.transforms.Resample()(waveform).repeat(3, 1, 1)
batch_consistency_test.py:        computed = torchaudio.transforms.Resample()(waveform.repeat(3, 1, 1))
compliance/utils.py:TEST_PREFIX = ['spec', 'fbank', 'mfcc', 'resample']
compliance_kaldi_test.py:    def test_resample_waveform(self):
compliance_kaldi_test.py:            output = kaldi.resample_waveform(sound, args[1], args[2])
compliance_kaldi_test.py:        self._compliance_test_helper(self.test2_filepath, 'resample', 32, 3, get_output_fn, atol=1e-2, rtol=1e-5)
compliance_kaldi_test.py:    def test_resample_waveform_upsample_size(self):
compliance_kaldi_test.py:        upsample_sound = kaldi.resample_waveform(self.test1_signal, self.test1_signal_sr, self.test1_signal_sr * 2)
compliance_kaldi_test.py:    def test_resample_waveform_downsample_size(self):
compliance_kaldi_test.py:        downsample_sound = kaldi.resample_waveform(self.test1_signal, self.test1_signal_sr, self.test1_signal_sr // 2)
compliance_kaldi_test.py:    def test_resample_waveform_identity_size(self):
compliance_kaldi_test.py:        downsample_sound = kaldi.resample_waveform(self.test1_signal, self.test1_signal_sr, self.test1_signal_sr)
compliance_kaldi_test.py:    def _test_resample_waveform_accuracy(self, up_scale_factor=None, down_scale_factor=None,
compliance_kaldi_test.py:        # resample the signal and compare it to the ground truth
compliance_kaldi_test.py:        estimate = kaldi.resample_waveform(sound, sample_rate, new_sample_rate).squeeze()
compliance_kaldi_test.py:    def test_resample_waveform_downsample_accuracy(self):
compliance_kaldi_test.py:            self._test_resample_waveform_accuracy(down_scale_factor=i * 2)
compliance_kaldi_test.py:    def test_resample_waveform_upsample_accuracy(self):
compliance_kaldi_test.py:            self._test_resample_waveform_accuracy(up_scale_factor=1.0 + i / 20.0)
compliance_kaldi_test.py:    def test_resample_waveform_multi_channel(self):
compliance_kaldi_test.py:        multi_sound_sampled = kaldi.resample_waveform(multi_sound, self.test1_signal_sr, self.test1_signal_sr // 2)
compliance_kaldi_test.py:            single_channel_sampled = kaldi.resample_waveform(single_channel, self.test1_signal_sr,
torchscript_consistency_impl.py:    def test_Resample(self):
torchscript_consistency_impl.py:        self._assert_consistency(T.Resample(float(sr1), float(sr2)), tensor)
transforms_test.py:    def test_resample_size(self):
transforms_test.py:        invalid_resample = torchaudio.transforms.Resample(sample_rate, upsample_rate, resampling_method='foo')
transforms_test.py:        self.assertRaises(ValueError, invalid_resample, waveform)
transforms_test.py:        upsample_resample = torchaudio.transforms.Resample(
transforms_test.py:        up_sampled = upsample_resample(waveform)
transforms_test.py:        downsample_resample = torchaudio.transforms.Resample(
transforms_test.py:        down_sampled = downsample_resample(waveform)

Do you have any thoughts on showing correctness? We could keep the previous algorithm in testing so that we can compare against.

[adefossez]

I'm usually testing against resampy, although the default resampy filters uses a different window than Hann, so with the default config, the delta is around 1%. I can look into what config to give it to have it use Hann window, so that the test is really robust.
This adds an external dependency for testing, but the advantage is cleaning up the torchaudio codebase.

[vincentqb]

I'm ok with an external dependency for testing. Knowing how well the current implementation in torchaudio match a library like this would be good to know.

[mogwai]

I'm not an expert in resampling by any means but I created a quick benchmark for speed and compared Log Spectral Distance and the Noise Ratio between slower more high quality methods in a notebook that might be useful for this thread

tldr; julius seems to be the best option

[adefossez]

I did a first pull request. As I adapted my code so that the resampling parameters match exactly those used before, I did not change the unit tests (which are comparing the resampling of one audio file to Kaldi original implementation). If you think it is worth it @vincentqb I can add extra unit tests to compare against resampy.

[vincentqb]

Thanks again for working on this :) I see #1087 as closing this issue. If this is not the case, please feel free to re-open, or if there are follow-ups, to open new issues.