spkrec-ecapa-voxceleb-mel-spec model modifies mel spectrum in place when used with CPU

[jacobjwebber]

Describe the bug

Thanks for your wonderful project. I would love to make my first speechbrain contribution and help fix this issue :). when I encode a mel spectrogram using a CPU using MelSpectrogramEncoder, the underlying mel spectrogram data is modified

Expected behaviour

Calculating the embedding should not modify the original data

To Reproduce

test_in_place.py

from speechbrain.pretrained import MelSpectrogramEncoder
import torch
from torch.nn.functional import mse_loss
import torchaudio

# On CPU
waveform, sr = torchaudio.load("tests/samples/TTS/LJ050-0131.wav")
waveform = torchaudio.functional.resample(waveform, sr, 16000)
spk_emb_encoder = MelSpectrogramEncoder.from_hparams(source="speechbrain/spkrec-ecapa-voxceleb-mel-spec", savedir="spk_emb_encoder_checkpoints", run_opts={"device": "cpu"})
mel = spk_emb_encoder.mel_spectogram(waveform)
mel_copy = torch.clone(mel)
spkr_emb = spk_emb_encoder.encode_mel_spectrogram(mel_copy)
print(mse_loss(mel, mel_copy))

# On CUDA device
waveform, sr = torchaudio.load("tests/samples/TTS/LJ050-0131.wav")
waveform = torchaudio.functional.resample(waveform, sr, 16000)
spk_emb_encoder = MelSpectrogramEncoder.from_hparams(source="speechbrain/spkrec-ecapa-voxceleb-mel-spec", savedir="spk_emb_encoder_checkpoints", run_opts={"device": "cuda"})
mel = spk_emb_encoder.mel_spectogram(waveform)
mel_copy = torch.clone(mel)
spkr_emb = spk_emb_encoder.encode_mel_spectrogram(mel_copy)
print(mse_loss(mel, mel_copy))

python tests/test_in_place.py
tensor(33.1424)
tensor(0.)

Environment Details

speechbrain '0.5.16'

Relevant Log Output

No response

Additional Context

It only happens with CPU

[asumagic]

Thanks for the convenient repro. I would note that this is not really specific to the CPU device: The bug seems to be in-place modification of the input tensor.

Since you're passing a CPU tensor in the second example, it gets migrated to CUDA and thus effectively cloned. If you .to("cuda") the waveform yourself the bug also reproduces.

[asumagic]

The bug is in InputNormalization when the norm_type is either "sentence" or "speaker", in which case it performs in-place assignments. I will make a PR soon.