Sound Cue Volume API

[grantjbutler]

TL;DR

It'd be great if NodeCG exposed a way to read and observe the volume value of an item in NodeCG's audio mixer for times where you need to roll your own audio playback.

Context

In developing a few bundles for a couple of different events and streams, I've come across a potential gap in the API surface for audio within NodeCG. While it's helpful that NodeCG exposes a way for doing audio playback, that's only for basic cases of audio playback. If you need to do something more complex, like using the Web Audio APIs, NodeCG can't handle this.

Normally, this is fine; I don't think I expect NodeCG to be able to handle every possible way someone could want to interact with audio in their bundle. Audio is a huge space with so many possible ways to build things that I think it's infeasible to try to cover all of that within NodeCG. However, once you make a decision that you need to use some of these other means of playing audio, you lose out on various aspects of integration with NodeCG, specifically integration with NodeCG's audio mixer.

Proposal

NodeCG exposes an API for observing the value of various volume faders in NodeCG's audio mixer. NodeCG already defines a number of internal replicants for the volumes of various groups (the master volume level and a bundle's volume level), as well as calculates a specific cue's volume level when playing a sound cue. While you can find these things within the NodeCG codebase to extract into your own bundle for observing, and extracting the cue's volume level calculation, you run the risk of your implementation going out of sync the the names of replicants changes, or if then algorithm for calculating volume level changes. Ideally, NodeCG should provide API for observing these values so it can retain control over various implementation details without a risk of breaking external consumers when the implementation changes.

Example Use Cases

Here are a couple of situations where I've run into this issue when I've wanted my custom audio playback to integrate with NodeCG's volume mixer and have needed to reach into the codebase to find and extract these various elements.

Personal Streams

For my personal streams, I play my intro and outro music through NodeCG using the Web Audio APIs. This gives me fine-grained control over the playback of the music, as I've built in looping sections for these music tracks. When playback reaches various points, certain chunks of the audio will start looping until I send a message to tell the graphic stop looping and let playback resume normally. I also have multiple versions of the same track playing in parallel to allow for transitioning between intensity in loops depending on sections of the song. This allows me to start playing my music, talk over them as I do my outro, and then when I'm finishing up my little speech, I press a button to let the music continue as I transition to my outro scene. Because I've built my audio in this way that doesn't leverage the basic audio playback features in NodeCG, I've lost the ability to integrate with NodeCG's audio mixer.

Charity Streams

For a few charity streams I'm involved in, they have the concept of "secret sounds". When a donation comes in, an audio cue is played to indicate that there's a donation. There are certain tiers of sounds (for example, every donation of $9.99 or below plays one sound, while every donation between, say, $24.99 and $10 plays another, etc), but there are also specific sounds associated with specific donation amounts (the "secret sounds"). Whenever a donation comes in, you want to make sure you've got the sound primed and ready for playback, so there's no delay in telling the sound to play and it actually playing.

You can build this with NodeCG in a couple ways. If you want to have every sound pre-loaded, then you need to create individual sound cues for each sound. This may not be possible, since these sounds are not known at the time of compiling your bundle and installing it into NodeCG; you could use NodeCG's asset handling to upload new sounds after you've installed your bundle. Even if you could register sound cues at runtime and not only in your bundle's package.json, then that means you've having one fader for each sound, which could result in the audio mixer in NodeCG getting super cluttered with 10s or 100s of sound cues, depending on how many sounds you want to play.

If you only want to have one fader for all sounds, then you need to move to swapping the audio file used for a cue dynamically when you get a donation. There's no public API for this functionality, either, so you need to dive into the internals of NodeCG again to figure out how to do this. Additionally, because you'd be telling the SoundJS instance to load a new audio file and are relying on the browser's caching mechanisms, then it could mean that you're waiting for the new sound file to load before you can show the donation.

If you instead manage all of the audio players yourself, you can avoid the swapping of sound cues and reloading of audio files because you can keep everything in memory. However, this approach loses the integration with the NodeCG audio mixer, so you can't control the audio level of these players from that screen of the dashboard. If NodeCG had an API for providing the volume level of various items in the mixer, then you can control the volume of all of these players with a single fader.