AudioBuffers remarks, questions and some proposals

[b-ma]

Hey, I open this issue because reading the code and trying to understand it, there are still some aspects that I struggle with and that I think could be clarified and maybe improved.

disclaimer: I just go through my understanding to try to be clear, I don't pretend anything is ground truth here

So, my impression is that there should be 3 different types of audio buffers while there are only 2 for now:

1. The buffers used to compute blocks of signal (i.e. alloc::AudioBuffer). Nothing special to say about this one except that it could be renamed to AudioBlock, AudioBus or AudioRenderQuantum to avoid confusion with the AudioBuffer exposed by the API (which would also lead to renaming BUFFER_SIZE to BLOCK_SIZE or RENDER_QUANTUM_SIZE for consistency)

2. The AudioBuffer as defined by the WebAudio API, which is only consumed by the AudioBufferSourceNode and the ConvolverNode or returned through a Promise by OfflineContext.startRendering. This one should be completely loaded in memory for very fast and accurate access by the nodes that consume it, audioContext.decodeAudioData should therefore work from a asset (e.g. some audio file) completely loaded in memory (retrieved from network (XHR call in JS land) or file system) and perform the following steps:

   • 1. decode to PCM data
   • 2. resample to the actual audioContext.sampleRate
   • 3. return an AudioBuffer that can be consumed without further processing by e.g. the AudioBufferSourceNode

3. Some other audio buffer which is more related to a streaming paradigm (data is received on the fly with questions of buffering, back pressure, etc.) and can be piped to some WebAudio nodes but is defined or related to other specs such as WebRTC (microphone, audio streams from network and MediaStreamAudioSourceNode) or HTML (<audio> tag and MediaElementAudioSourceNode). In that case decoding and resampling can only be done on the fly when the chunks are received, and the WebAudio nodes do not expose any start, stop or seek possibilities.

In the current implementation, my impression is that the AudioBuffer defined buffer.rs try to handle both 2. and 3. paradigms in the same way and, is finally more close to 3. and not really optimized for 2.

Do this makes sens or am I mistaking somewhere here? If you think it's a good idea, I would be happy to try to propose something really dedicated to 2.

Related issues

• Make AudioBuffer more compliant #30
• Add DecodeAudioData to AudioContext #42

[orottier]

Hey, thanks for thinking along, it's much appreciated. I think you have summarized the three cases very well.

I fully agree with 1) The alloc/AudioBuffer should be named better to avoid confusion with the user facing AudioBuffers. alloc is a really bad mod name too. There should probably be a render mod containing the Graph, AudioProcessor and the new AudioRenderQuantum

I'm interested to hear your proposal for a distinction of 2 and 3.
In my eyes, a memory resident buffer is the buffer::AudioBuffer and anything that acts as a continuous source of audio is an Iterator<Item=AudioBuffer>.
For example, if you decode an Ogg Vorbis file, you receive chunks of audio, fitting this iterator style. Decoding a wav (currently) works sample-for-sample so we just take BUFFER_SIZEd chunks.

In any case, please let me know what you intend to refactor. We can surely work out something that is better than the current state

[b-ma]

Ok cool, thanks for your answer. What I can imagine to advance on that is:

• First, make a PR with the renaming stuff and file reorganisation, which is only refactoring and should not impact any behaviour.
• For the rest, I really think it's feasible to work on something focusing on 2. alongside the existing code and without interfering with the other modules and tests. Then, if I manage to make something work at some point, we can just discuss / amend the result, and eventually see how to integrate it properly.

Sounds like a plan?

[orottier]

Sounds good to me!