`hann_window` does not obey COLA; therefore, it cannot be used with `istft`

[PetrochukM]

🐛 Bug

In order to enable inversion of an STFT via the inverse STFT in istft, it is sufficient that the signal windowing obeys the constraint of “Constant OverLap Add” (COLA). This ensures that every point in the input data is equally weighted, thereby avoiding aliasing and allowing full reconstruction.
https://docs.scipy.org/doc/scipy/reference/generated/scipy.signal.check_COLA.html

Learn more: https://ccrma.stanford.edu/~jos/sasp/Mathematical_Definition_STFT.html#19930

To Reproduce

>>> from scipy import signal
>>> import torch
>>> signal.check_COLA(signal.hann(1024, sym=False), 1024, 1024 - 256)
True
>>> signal.check_COLA(torch.hann_window(1024).numpy(), 1024, 1024 - 256)
False
>>> window = torch.hann_window(1024).numpy()
>>> window[0] *= 0.5
>>> window[-1] *= 0.5
>>> signal.check_COLA(window, 1024, 1024 - 256)
False

Expected behavior

The expected behavior is that the hann window is implemented such that it is useable with istft.

[PetrochukM]

While it looks like the windows are mostly similar after visualization, it does look like the PyTorch window is off scipy on the third decimal...

from scipy import signal
import torch

window = 1024
torch_hann_window = torch.hann_window(window)
scipy_signal_window = signal.hann(1024, sym=False)
print('scipy_signal_window', scipy_signal_window[-1])
print('torch_hann_window', torch_hann_window[-1])

scipy_signal_window 9.412358699445456e-06
torch_hann_window tensor(9.4175e-06)

The scipy window is the one Librosa uses under the hood: https://librosa.github.io/librosa/_modules/librosa/filters.html#get_window

[vincentqb]

What do you get if you use that window in istft? :)

[PetrochukM]

Perfect reconstruction is not possible and aliasing occurs.

Sources:
https://docs.scipy.org/doc/scipy/reference/generated/scipy.signal.check_COLA.html
https://www.dsprelated.com/freebooks/sasp/Constant_Overlap_Add_COLA_Cases.html
https://www.dsprelated.com/freebooks/sasp/Dual_Constant_Overlap_Add.html

[vincentqb]

I meant: does torchaudio silently computes the wrong quantity, gives an error, returns NaNs, etc?

[PetrochukM]

Uhm. Sorry. There is no explicit error or problem with ISTFT. The problem is that the window implementation prevents STFT -> ISTFT from giving a perfect reconstruction and causes aliasing.

[vincentqb]

I see. How should the window be modified to satisfy the COLA condition? Which windows support COLA and not in torchaudio?

[faroit]

@PetrochukM Interesting

The expected behavior is that the hann window is implemented such that it is useable with istft.

• Are you sure this has an impact in practice?
• have you checked to lower the tolerance of the COLA test? If it doesn't pass because the default tolerance is 1^-10, I think this can be relaxed.
• I got an RMS of 10^-9 for random signals with a half-overlap hann window in this test scripts

[PetrochukM]

Thanks for taking the time to help! I do wish both of you would read my initial comments more thoroughly as they already answer your questions. Let me know if I could have made it more clear. Thanks for contributing your time to open source!

@vincentqb To answer your question... scipy.signal.hann(1024, sym=False) satisfies the COLA condition which is also used by librosa. (See the first comment)

I suppose the right modification would be to copy scipy and to test for COLA.

@faroit The window is off by a minimum of 5e-3. That is a large difference, in my opinion. We're not talking about 1^-10. (See the second comment)

[vincentqb]

Thanks for taking the time to help! I do wish both of you would read my initial comments more thoroughly as they already answer your questions. Let me know if I could have made it more clear. Thanks for contributing your time to open source!

@vincentqb To answer your question... scipy.signal.hann(1024, sym=False) satisfies the COLA condition which is also used by librosa. (See the first comment)

I suppose the right modification would be to copy scipy and to test for COLA.

Sorry for the misunderstandings, and thanks again for your contributions :) What I meant with the first question is: would you be willing to open a draft pull request in torch with the goal of highlighting the exact changes that need to be done to make the hann window satisfy the COLA condition? If I'm missing your message with this information, please do link it so I can see the diff, thanks again for the information!

Given that these windows live in torch and not torchaudio and changing them would modify what the users would get as output, we likely won't be able to modify them directly. We could discuss a proposal to have an alternate COLA-compliant form of each to make sure we minimize this impact. :) We could also add that information to their documentation.

For the second question, I meant: could you check which of the other windows that pytorch has (e.g. torch.*_window in here) satisfy or not COLA, and which we would expect to satisfy COLA or not? Again, it may be worth adding this to the documentation of each. Am I also missing this information from your earlier comments? Thanks again!

[f0k]

This seems to be a precision issue:

>>> import scipy.signal
>>> import torch
>>> w1 = scipy.signal.hann(1024, sym=False)
>>> w2 = torch.hann(1024)
>>> w3 = torch.hann(1024, dtype=torch.float64)
>>> scipy.signal.check_COLA(w1, 1024, 512)
True
>>> scipy.signal.check_COLA(w2.numpy(), 1024, 512)
False
>>> scipy.signal.check_COLA(w3.numpy(), 1024, 512)
True

So it works when letting Pytorch compute the window in double precision like scipy. It also works when converting back to single precision, for both:

>>> scipy.signal.check_COLA(w1.astype(np.float32), 1024, 512)
True
>>> scipy.signal.check_COLA(w3.numpy().astype(np.float32), 1024, 512)
True

I suppose the right modification would be to copy scipy and to test for COLA.

The scipy code is here: https://github.com/scipy/scipy/blob/6d576be/scipy/signal/windows/windows.py#L109-L118, called with a=(0.5, 0.5).
The Pytorch code is here: https://github.com/pytorch/pytorch/blob/bca8280/aten/src/ATen/native/TensorFactories.cpp#L894-L899, called with alpha=0.5, beta=0.5.
Not sure if there's any way to get more accurate results in single precision? Otherwise what we could do in torchaudio would be to create the window in double precision and convert it to the target dtype afterwards.

@faroit The window is off by a minimum of 5e-3. That is a large difference, in my opinion. We're not talking about 1^-10. (See the second comment)

@PetrochukM: It's 5-e9 in your second comment. But that's for the last coefficient; the maximum difference is around 1.2e-7. In any case, there's a way to pass scipy's COLA test with a float32 window, but not with the one produced by torch.hann_window(1024), so we could improve things a little.

[PetrochukM]

@f0k Thanks for looking into this! Also, I was totally wrong with the maximum difference, thanks for correcting me!

It sounds like there is a little bit of room for improvement. It also sounds like there is still room in replicating librosa and scripy's behavior.