Inter-app audio (Consistent request from ISVs, a la VSTs, Audio Units, Rack Effects)

[cwilso]

Likely a stream source/sink.

[ciprianimike]

this is exactly what i was waiting for

[idanbeck]

I'd like to make a serious case for supporting at minimum VSTs (if not the other architectures) in Web Audio such that plug ins could be loaded by Web Audio applications in the browser.

We are undergoing an effort to develop a fully feature in-browser music creation suite, or DAW, that will allow anyone to produce music in the cloud. Our hope is to bring this application to a level of competitive functionality and performance with other DAW options like Logic, Ableton, or Garageband and one of the most critical features that we've come across will be the support of plug ins.

This feedback primarily comes from the discussions we have from musicians and producers as we're showing them early demos and examples of our application in action. While it's clear Web Audio will be more than sufficiently performant to achieve some equivalent processing in the browser, it's now also clear that it will be possible to create an application with sufficient performance and functionality on the web development side of things as well in Chrome. There are some sticky bits, that will likely continuously improve as the platform continues to evolve, but overall the path is relatively clear with what will likely be small work arounds needed for certain things.

However, one thing that there is no clear work around for is the support of 3rd party plug ins for audio, and being that many musicians and producers have amassed a large collection of such plug ins and tools, we feel that it will be critical to enable these plug ins within any given music creation suite even if it is on the web. The alternative would be to go to all of the plug in developers and convince them to port their plug ins to web audio, and further it would also require their agreement with regards to a plug in standard to be used with Web Audio applications. Such agreement will not only be hard to enforce, but also very difficult to reach.

I feel that the support of VSTs or other such plug in architectures with Web Audio will be a must if Web Audio is to get to a point where fully featured music creation applications can be built in the browser. This has a lot more to do with adoption than the capability of building equivalent features/performance in Web Audio directly.

One option that comes to mind is to host a plug ins within a secure process that Web Audio has access to as a given audio node. This way Web Audio plug in nodes could be created within a sandbox where only certain OS actions are permitted specific to audio and the display of the plug in interface. A really exciting implementation would render the plug in interface directly into a canvas element for example.

Sorry for the verbose comment, but we feel very strongly as we're making a pretty big bet that music production could be accomplished entirely in the web. The support of plug ins, from this perspective, is really a stop gap solution to encourage adoption. However, without this stop gap applications built in web audio will always be hindered in comparison to their desktop counterparts.

[padenot]

Running VST as plugins in the browser seems unlikely to happen.

[cwilso]

Why? Providing access to VSTs as a collection of streams or something like
that seems doable.

On Tue, Oct 21, 2014 at 3:58 AM, Paul ADENOT notifications@github.com
wrote:

Running VST as plugins in the browser seems unlikely to happen.

—
Reply to this email directly or view it on GitHub
#358 (comment)
.

[padenot]

hrm, that means running untrusted x86 code in a web page, or am
I missing something?

[cwilso]

I didn't say it would be automatically safe, or better than NPAPI, ActiveX, etc.

Yeah, I'd prefer a asm.js style approach with registrations. But any musician today likely has a very large amount invested in VST instruments, and it's the communication system used by many pieces of software.

[stuartmemo]

What do you imagine the use case to be here? Is it the api talking to a compiled VST in the browser, or exposing an input/output stream to applications outside it?

[jernoble]

We object to adding a plugin system to WebAudio. The entire point of HTML5 in general and the WebAudio API in specific is to move away from platform plugins.

[jussi-kalliokoski]

While I can agree with the potential of this, the only way this could ever work reasonably is by having virtual devices and an external DAW connecting to them. That would also not restrict this feature to VSTs. However, VSTs also often expose parameters that are not controllable by MIDI nor signals, but only through the API. Just a stream API won't suffice for most cases, and to get the full benefit you'd probably want to expose the UIs somehow as well and ughhh plugin containers for legacy x86 VSTs. This would also be of zero use to mobile or linux users (saying as someone who has spent countless time trying to get even simple VSTs running on linux, because most VSTs not only depend on x86 but also on OS APIs). I say the ball is in the manufacturers' court to introduce web audio as a cross-compilation target.

[stuartmemo]

I say the ball is in the manufacturers' court to introduce web audio as a cross-compilation target.

Agreed.

I'm having a hard time seeing the actual user benefit in this at all - it seems extremely messy. I get why existing VST/AU authors would like this to prevent rewrites, but as a user I have no interest whatsoever in connecting my web audio apps to such things. I'd like web audio to evolve to be its own thing, not be held back wasting time worrying about old desktop software.

[notthetup]

This somehow feel against the grain of the WebPlatform to me. The idea of HTML5 and WebAudio was to enable a plugin-free Web. But this brings us back to plugin compatibility issues, 32/64bit compatibility, native UI rendering issues, OS API compatibility and other things that the web platform is trying to move away from.

And to @jussi-kalliokoski's point there exist toolchains like emscripten which can be used to have the JS as a compile target for plugins. Libraries needed to interface between standard WebAudio and VST type interfaces can be done in JS.

The only argument I see for this is performance, and that maybe some of the computation done natively in VSTs can't be done in WebAudio (using AudioWorker) in real time. But maybe that's something that needs to be tested and measured before we can assume it's true.

[cwilso]

I'd like the set of people having violent allergic reactions here to calm down a bit.

We have core use case (http://www.w3.org/TR/webaudio-usecases/) scenarios around 1) online music production tool, 2) music creation environment and 3) connected DJ booth. If, in the end, we are not providing for those scenarios as well as every native platform out there, then I feel we have failed in our mission. If you disagree with the mission of "make a platform for music tools that fulfills user needs as well as or better than native platforms," we should hash that out now.

On investigating the state of the art in music production software today, and speaking directly with many of the manufacturers of them, it became clear to me these tools are HEAVILY entwined with third party systems - this is an interconnected ecosystem of software from multiple vendors working together. I'm by no means an expert in these technologies, but the ecosystem here is huge.

There are multiple entwined sub-scenarios implied by our mission:
a) connection of multiple independent pieces of software from different vendors, all participating in the same audio production - e.g. using DJ software and a soft synth/sequencer synced together, outputting to the same audio device
b) the direct "insert" style of the former, e.g. inserting a guitar amp simulator effect module produced by a third party inside a production tool created by some other party,
c) specifically, the very large corpus of VST/AudioUnit/etc that are already invested in by users, who are going to be naturally resistant to switching to web production tools if it means there is no way to utilize their current investments in third party tools.

On a), we have not looked at this yet, and I think we need to at least have some ideas of how one might do this - passing audio pipes through threads, to different "apps", etc. Apple directly added support for inter-app audio to iOS7 over a year ago (http://9to5mac.com/2013/06/12/ios-7s-new-inter-app-audio-introduces-universal-audio-routing-between-apps/) after the success of JACK and AudioBus, so one can hardly claim this is not mainstream. Producers of DAW-type software that I've spoken with (e.g. Idan) have universally asked about communication with other software tools, because no one expects users to stay only within the confines of their application.

b) may just be a subset of a), I don't know - except it does have some implications in the audio graph and latency, as previously discussed.

Specific on point c), musicians, producers, DJs et al typically have a lot invested in their set of instrument and effects, so I'd like us to not be knee-jerk dismissive of accessing VSTs without understanding the adoption cost to audio apps on the web. @stuartmemo, saying "but as a user I have no interest whatsoever in connecting my web audio apps to such things" leads me to believe you have no personal investment in VSTs - because as a user who's dropped a couple of thousand dollars in instruments and effects, I can't imagine not having at least a passing interest in being able to access that investment in web tools.

I don't know what the right answer is here. I do not believe we are introducing "32/64bit compatibility issues" et al; I do understand (sheesh, people, I worked on IE for 15 years) that having any system that relies on something installed on the machine it's being run on has fragility concerns. Those are concerns, would need to be investigated and understood, etc.

I do believe that expecting emscripten to flat-out convert VSTs is going to be a hideous ball of something very unpleasant - not just because of the 50% perf hit. I additionally know, however, that not all plugin architectures are like that; Propellerheads' RackEffects, for example, are essentially in IL, and I'm reliably informed recompiling them into JS is probably not hard at all.

To the point in general - "plugin" was meant to relate specifically to the scenario, and not necessarily to a specific native code API. It's all very well to say "the ball is in the manufacturers' court to introduce web audio as a cross-compilation target", but we are not giving manufacturers anywhere near the tools they would need to even have that compilation target in terms of packaging and communication - which is why I filed an issue.

[jussi-kalliokoski]

but we are not giving manufacturers anywhere near the tools they would need to even have that compilation target in terms of packaging and communication - which is why I filed an issue.

Alright, fair enough, I'm game. So let's list what we need WRT being a viable compilation target. Here's what comes off the top of my head (maybe we already have some of these, fill me in on it, also some are not strictly necessary):

• Multiple IO for AudioWorkers (or being able to connect to a specific channel).
• Reading the IO latency.
• IO change (connect/disconnect) notifications. This can be done by the host application though, we don't necessarily need to - and probably shouldn't - add support for this on our side.
• Async output for AudioWorkers.
• I'm not sure how widely this is supported / used in practice, but, changing the number of available inputs and outputs after instantiation.

Adding these would probably cover a significant portion of the cases that don't touch OS APIs. I've never looked at the GUI side of things for VST so I have no idea if there's something missing from that side.

Obviously, the web being a viable target for compilation is not limited to Web Audio API - we also probably need to define the common JS APIs that the various hosts and plugins need to implement in order to have interoperability between them.

[joeberkovitz]

Chair hat on: Regarding the V1/TPAC tag for this issue: I think this is a reasonable issue to discuss at the F2F in the spirit of, "let's make sure V1 doesn't rule this out". However, just because there's a use case for online music production tools (I think I drafted it, actually :-) doesn't mean we have to introduce support for every aspect of that use case in the initial rollout of Web Audio. We need to avoid V1 turning into V2 by way of feature creep.

If the V1 Web Audio API supports the creation of pure-web online music production tools now, but permits future bridging to VSTs later assuming there is agreement on that, I think we'll have achieved a worthy goal.

Chair hat off: I have mixed feelings here. I am one of those who have dropped a lot of money on VSTs in native environments because they are powerful musical tools, and I need them for my work. But I want to see the emergence of pure-web equivalents to VSTs; it would be kind of sad if web-based DAWs just wound up being shells around native plugins, and anyone wanting to access a project in one of those DAWs had to have totally compatible plugins (does this remind anyone of the not so distant past on the web in general?)

[cwilso]

Changed the title, because everyone seems to be fixating on "plugins 'r bad, mmmkay" here. This is about creating music tools that can work together inside the platform, more than it is about a particular plugin hook.

I was surprised at the prevalence and dependence on plugins in today's music app ecosystem. I do not think Web Audio v1 fully supports the "creation of pure-web online music production tools" today, because it do not provide any way to enable apps to work together aside from direct unprotected JS inclusion - and I can't imagine it's a good idea to consider enabling users to insert JS from third-party vendors into your DAW. Considering that even on iOS, with its single-app-at-a-time user experience - inter-app audio is a major thing, I think this is important to at least discuss the implications of, since I think it would be very sad if the web platform never took off as a music platform because in essence, you can only use one tool at a time.

Separate from this is VSTs, et al and whether they should be accessible from the web.

[joeberkovitz]

I didn't see any title change take effect, but I think this shift in focus is clarifying and helpful. I'm not entirely clear what the definition of "app" would be in a goal statement of "we want multiple apps to communicate", so that might be a good first step in this discussion.

It also feels to me that any highly modular, processing-based application domain (not just DAWs, e.g. image processing with filters etc.) are affected by the same questions regarding unprotected JS inclusion and robustness.

Anyway, it will be good to devote some discussion to this at TPAC and make sure we aren't screwing up the future. We can also network with other groups and see if there are parallel issues and solutions being discussed.

BTW I wonder if this isn't connected to some of the inter-audio-context issues raised in #257 -- that seems somewhat relevant to the idea of inter-app audio.

[jussi-kalliokoski]

FWIW, I agree that inter-app audio is a use case we need to address sooner or later.

[cwilso]

Darn separate "save" button for title edits. Grr.

[cwilso]

#257 is really about intRA-app audio - or more to the point, how and

[joeberkovitz]

Need clearer definition of what developers want, and why, and what they'd be willing to live with.

[mdjp]

A previous meeting action was to verify the use cases (http://www.w3.org/TR/webaudio-usecases/)
I believe that the following use cases could have a requirement for inter app audio.

2.3 Online Music Production Tool.
Dependency on inter app audio: High assuming the requirement is to recreate DAW functionality.
2.4 Online Radio Broadcast.
Dependency on inter app audio: Medium, routing of audio to existing effects may be a requirement.
2.5 Music Creation Environment with Sampled Instruments.
Dependency on inter app audio: High
** Edit - not dependant on inter app audio 2.7 Playful sonification of user interfaces.
Dependency on inter app audio: High Edit **
2.10 Web based guitar practice service.
Dependency on inter app audio: Medium

We can of course support these use cases without inter app audio however it would have an impact on their usefulness ( in my opinion.)

[jussi-kalliokoski]

On 9 Feb 2015 12:56, "Matthew Paradis" notifications@github.com wrote:

2.7 Playful sonification of user interfaces.
Dependency on inter app audio: High

Could you elaborate on how this has a high dependency? :)

[mdjp]

Could you elaborate on how this has a high dependency? :)

This may be my own interpretation of the use-case, but the ability to pass audio (and control data) between environments is fairly common for this type of application. In addition third party synthesis or signal processing may be required (VST instruments etc). I should be clear that in my case a user interface could be a physical connected object as well as a graphical ui.

[jussi-kalliokoski]

(edited our comments to rid of the email noise I created :)

@mdjp That's a whole different beast, though. Let's look at a few scenarios:

1. An app adding sound effects to its own UI.

In this scenario, the app itself decides what kind of sonification to apply. In here, third party synthesis such as VST could be used, but there's no specific justification here that doesn't apply to pretty much any use case we can think of.

2. An app adding key-stroke/click whatnot sounds to all applications globally in the OS.

In this scenario, there's no need for inter-app audio per se, but more something like global keylogging and pointer tracking capabilities (color me frightened), which is outside the scope of this WG.

3. An app adding sonification to the UI of an external device.

In this scenario, we're transmitting audio and control data across devices, so this is more in the domain of Web MIDI API and WebRTC.

[mdjp]

Completely agree with the first 2 points. I think that the context that I was referring to is actually very different and should not be considered in the same use-case. I have made a note in the original comment to reflect that.

[mdjp]

No considered for V2, use WASM/Worklet/Audio Device Client