Customizable windowing function for Analyser node

[rtoy]

The AnalyserNode currently always windows the signal using a Blackman window. It would be nice if other windows could be supported so that the user can tradeoff the smoothing from the window and the spectral resolution.

At the very least it's useful if no window (also known as rectangular) were allowed to be specified.

[padenot]

Virtual F2F:

• This can be done in a backwards-compatible way
• We need to figure out what authors need (there are a bunch of window functions)
• We need to re-check the viability of WASM-based FFTs in terms of speed (now that SIMD is around the corner in terms of implementation shipping), for custom processing, but changing the windowing function is common

[padenot]

Proposal:

enum WindowFunction {
  "blackman",
  "rectangular"
};

partial interface AnalyserNode {
  attribute WindowFunction windowFunction;
};

From the call, those two are the most important, but we could add others.

An AnalyserNode is starting in "blackman" mode initially.

[padenot]

Something to consider is what to invalidate when changing the mode:

• Is it better to start over from the time domain data, and recompute everything from scratch
• Is it acceptable to smooth, say, a blackman window and a rectangular window? It's technically wrong, but save an FFT.

[rtoy]

See https://en.wikipedia.org/wiki/Window_function for other possibilities.

[khoin]

Gaussian is a popular one.

Kaiser window seems to be a very flexible one, but I don’t know if this is something that should be out-of-the-box for WebAudio.

I did short summaries of characteristics and usage of the some of the Windows in this repo: https://github.com/khoin/ExtraWindows

[khoin]

I take that back. I meant Hann and Hamming are popular ones, not Gaussian.

[rtoy]

Heh. I have no objections to adding Hann and Hamming windows.

I think we'll probably want to avoid window functions that have parameters controlling the window function. It's not impossible, but it is kind of hard to come up with a nice way to specify the window and the appropriate parameters.

[khoin]

I agree with you there.

Doesn’t seem like something WebAudio should handle.

Providing a way to customize windows (like PeriodicWave) should be enough for devs to utilize the smoothingTimeConstant, min/maxDecibels, binCount (like how they'd utilize custom PeriodicWave with OscillatorNode)

[rtoy]

Do you mean essentially a method that takes a set of Fourier coefficients of the window function? Then the Analyser would do an IFFT of the appropriate size (or interpolate) to get the window function of the correct size?

That would also be acceptable to me. But I wonder how much it would be used. I don't recall anyone really asking for new window except rectangular.

[khoin]

No. I didn't mean that. As far as I know, I have not seen anyone define/generate windowing functions using Fourier coefficients. If there are, they would be the Blackman window families, which are weighted cosines.

I meant that the benefits of allowing custom windows is akin to the benefits of custom PeriodicWave where people can reap the stuff that comes with OscillatorNode.

I imagine custom windows for AnalyserNode to look something like this:

let a = aC.createAnalyser(); // default fftSize = 2048 (spec)

a.minDecibels = -120;
a.maxDecibles = 0;
a.timeSmoothingConstant = 0.5;

let win = new Float32Array(a.fftSize);
// win.fill(1);  // Rectangular
win = win.map((x,i) => Math.abs(1 - ((i/a.fftSize) - 0.5))); // Barlett (triangle)

a.setWindowingFunction(win); //throws error if win does not match a.fftSize
// a.setWindowFunction ?
// a.setWindow ?

// extending @padenot's proposal
a.windowFunction = "custom";
// Comparing to OscillatorNode.type = "custom"; // throws Invalid State Error if no custom window is set.

[khoin]

I just realized that something needs to be done if a.fftSize is changed... Any thoughts?

I don't know how the change of a.fftSize is currently being handled in browsers or if the spec says something about it. Do you guys regenerate your Blackman window?

Spec says:

If the fftSize is changed to a different value, then all state associated with smoothing of the frequency data (for getByteFrequencyData() and getFloatFrequencyData()) is reset.

And also here:
https://webaudio.github.io/web-audio-api/#fft-windowing-and-smoothing-over-time

but spec doesn't say how windowing is handled when fftSize changed

[rtoy]

I'm not opposed to the idea, but I wonder how often it would be used. It adds a bit of complexity for something that may not be used.

Yeah, I was thinking about what happens when the fftsize changes. Resample the given waveform? Throw an error? Silently fallback to Blackman?

Most of the time I see the Analyser used for visualization and not really analysis. In that case the given windows are ok. For more complicated analysis, perhaps a ScriptProcessor or AudioWorklet is the right answer.

[khoin]

I think custom windows to be handled by WebAudio should not be considered if the majority of use case (do we have surveys?) are visualization. Perhaps rectangular, Han, Blackman is enough.

If someone wants to do analysis, they’d be building their own AudioNode already. I mean... there’s lots of consider — padding, even-odd (periodicity/symmetry), etc. Custom windowing support would appear to be a very narrow use case.

[rtoy]

Sounds good to me. Just the two suggested plus Hann and Hamming. No custom windows.

Creating a survey is easy. Getting it out to enough people might be hard.

But if someone has a solid use case where this is really important, we'd probably consider it. Please speak up if anyone has a solid use case for this!

[rtoy]

Note that https://en.wikipedia.org/wiki/Window_function#Hann_and_Hamming_windows says there's more than one Hamming window. We'd have to arbitrarily pick one of them.

[khoin]

I can't find the paper where Hamming proposes 25/46. However, if Hamming intended the window to minimize the peak of the largest side-lobe, then perhaps we should pick the coefficient that best matches that intention (i.e. 0.53836 instead of 0.54).

comparing JOS: https://ccrma.stanford.edu/~jos/sasp/Hamming_Window.html

[rtoy]

I would guess for our purposes any of them would work. We just need to pick one. We could, I suppose, allow both windows, but that's probably not necessary.

[padenot]

AudioCG meeting:

• having a buffer for this seems wasteful, but it would work, modulo setting the fftSize, but there is precedent in the API with setPeriodicWave/type with inverse fft for the wave table vs. simple waveforms
• simplicity is nice. rectangular, hamming, hann go a long way

[padenot]

Resolution:

• Something like https://github.com/WebAudio/web-audio-api-v2/issues/19#issuecomment-649692481
• "blackman" (the default), changeable to "rectangular", "hamming", "hann"

[rtoy]

TPAC 2020:

Does the window need to be changed? If not, then perhaps the best approach is not to allow the window to be changed. The we don't need to define what happens.

[rtoy]

Virtual F2F 2021: Support the types listed in https://github.com/WebAudio/web-audio-api-v2/issues/19#issuecomment-656237528

Also, the window will be changeable. When it is changed, everything is reset like what happens if you change fftSize. Exact API to be determined still.