OfflineAudioContext needs a way to handle audio of arbitrary duration

[olivierthereaux]

Originally reported on W3C Bugzilla ISSUE-21311 Sat, 16 Mar 2013 17:58:03 GMT
Reported by Joe Berkovitz / NF
Assigned to

Reference from mailing list:
post: http://lists.w3.org/Archives/Public/public-audio/2013JanMar/0395.html
author: Russell McClellan russell@motu.com

"[OfflineAudioContext] really should provide some way to receive data block-by-block rather than in a single "oncomplete" callback. Otherwise, the memory footprint grows quite quickly with the rendering time. I don't think this would a major burden to implementors, and it would make the API tremendously more useful. Currently it's just not feasible to mix down even a minute or so. If this is ever going to be used for musical applications, this has to change."

Chris Rogers stated in teleconference 14 Mar 2013 that it is in fact feasible to mix down typical track lengths of several minutes with the single oncomplete call. A discussion of block size suggested that any breaking of audio rendering into chunks should be fairly large to avoid overhead of switching threads and passing data.

[olivierthereaux]

Original comment by Jussi Kalliokoski on W3C Bugzilla. Sat, 16 Mar 2013 18:49:14 GMT

Chris Rogers stated in teleconference 14 Mar 2013 that it is in fact feasible to mix down typical track lengths of several minutes with the single oncomplete call.

One thing is that situations where you might expect to get long renderings (several hours) are studio software and scientific applications. Neither are typically run on devices with little memory, and if you think that uncompressed stereo CD quality gives you 80min @ 700MB, the problem may or may not be very serious. Either way, I think that the offline context should have an error callback as well, and if you run out of memory during the rendering the callback should be invoked.

However, to speculate a bit, another approach for handling long renderings is to direct the output to a MediaStreamRecorder which will hopefully in the future allow extracting compressed formats as well. This would allow the UA take care of the render/compress cycle and block sizes, and the API would just hand the developer a readily rendered blob to use however needed. I'm not quite up to date on MediaStreamRecorder though, perhaps Roc knows more about whether this is feasible or not.

[olivierthereaux]

Original comment by Robert O'Callahan (Mozilla) on W3C Bugzilla. Sun, 17 Mar 2013 22:57:19 GMT

(In reply to comment #1)

However, to speculate a bit, another approach for handling long renderings
is to direct the output to a MediaStreamRecorder which will hopefully in the
future allow extracting compressed formats as well.

The whole point of MediaStreamRecorder is to support compression so I think that'll be supported as soon as MediaStreamRecorder is implemented.

This would allow the UA
take care of the render/compress cycle and block sizes, and the API would
just hand the developer a readily rendered blob to use however needed. I'm
not quite up to date on MediaStreamRecorder though, perhaps Roc knows more
about whether this is feasible or not.

It's feasible but we need to specify how MediaStreamRecorder and OfflineAudioContext interact since currently MediaStreams are real-time-only.

I think it's pretty important to have some mechanism whereby OfflineAudioContext can deliver data incrementally.

[bjornm]

Hello everyone,

I believe incremental delivery would be very useful, especially in combination with #317 suspend() / resume().

To add to the discussion, please let me describe my real world use case, a DAW (Soundtrap). Some of you might feel it is kind of a narrow use case, but I hope not all of you :) I'm rendering 10+ audio tracks with heavy effects processing, in real-time. Only one or two of the tracks have low-latency (live) requirements coming from getUserMedia (microphone), the other tracks are pre-recorded audio or midi-tracks. After heavy optimization we can still see occasional performance problems after a few tracks, depending on effects and hardware. Command line "top" shows that all but one cpu core is basically idle - AudioContext is mostly single-threaded.

So, I would like to utilize all cpu cores better, and I can make a trade-off between low-latency and parallel processing. Maybe one way to do this would be to spread the tracks across multiple OfflineAudioContexts and use incremental delivery to render small buffers, just in time. For this to work I'm assuming that OfflineAudioContexts get scheduled to run on any available cpu core by the OS. Then, the delivered buffers would be appended to the live AudioContext somehow. Probably some part of the processing would be unified in that context for efficiency reasons, such as master reverb and volume.

An example of how this API could be implemented - more or less using existing or proposed methods:

To create an offline context with incremental delivery, an additional parameter could be added to the constructor:

var ctx = new OfflineAudioContext(numChannels, numFrames, sampleRate, chunkSize);

Every chunkSize frames you get a callback:

ctx.oncomplete = function(...) ...

To throttle processing you can use:

ctx.suspend()
ctx.resume()

It would be convenient it numFrames could be null. This would make the OfflineAudioContext keep processing forever until release is called:

ctx.release()

This should allow for a huge performance improvement. I'm guessing 4-8x throughput for a modern computer. Furthermore, it would become less sensitive to glitching, since parallel paths could have bigger buffers.

Regarding memory, I believe that users will run "studio software" on pretty much any device which allows for it - mobile, tablets and desktops. We run on iPad and it's pretty picky about allocating memory e.g. when creating OfflineAudioContexts or AudioBuffers.

Please see this as an example for the purpose of discussion. I'm speculating here and if there are other ways to do this that have already been discussed, even better!

Thanks,
Björn

[cwilso]

#302 is the current capture of design constraints.