AudioSource & AudioListener

[arakash92]

I wanted to have certain sounds in the game sound as though they are further away, so I implemented two classes, ig.AudioSource and ig.AudioListener.

I could add this to plusplus and submit a pull request?

Consider a crate that plays a 'break' sound.

EntityCrate = ig.global.EntityCrate = ig.EntityExtended.extend({
    init: function() {
        this.audio = new ig.AudioSource();
        this.audio.sounds.break = new ig.Sound('path/to/sound.*');
        this.audio.suonds.break.volume = 0.5;
    },

    update: function() {
        this.parent();
        this.audio.pos.x = this.pos.x;
        this.audio.pos.y = this.pos.y;
    },

    break: function() {        
        //play sound
        this.audio.pos.x = this.pos.x;
        this.audio.pos.y = this.pos.y;
        this.audio.break.play();
    }
});

Now, you probably want to have the AudioListener at the camera's position.
So, in your game's init function:

init: function() {
    this.audiolistener = new ig.AudioListener();
},

update: function() {
    //update the position of the audiolistener
    this.audiolistener.pos.x = this.camera.pos.x;
    this.audiolistener.pos.y = this.camera.pos.y;
},

Notes

*You can still play sounds outside the context of the game world by simply playing ig.Sounds (not via an AudioSource object) for this like UI.
*You can set the radius of an AudioSource which determines how far away its sound will be heard.
*All sound objects still have a functional volume level. This is taken into account.
*I also have other improvements on the Sound class which allows for easy looping (via pause(), unPause() and continue() methods)
*I am investigating making use of the new WebAudio API to try and create panning for directional sound.

[collinhover]

Cool! This would be a great addition. Some questions:

• Why not just use the camera as the audio listener, instead of adding extra overhead of the audio listener? Sounds can refer to the camera via ig.game.camera and set volume/stereo/etc based on their position relative to the camera.
• What is the benefit of using ig.AudioSource vs just extending or injecting ig.Sound?

Also, ping @bdaenen, he mentioned he was looking into something similar.

[arakash92]

I'd like it to be modular, you might want to have multiple audio listeners. One for the camera and one for player perahps.
Consider the player being in a car, all sounds being listened to from that position might be lower than the where the camera is. Sure, you could move the listeners position etc but it's nice to have that modularity.

I decided to not inject ig.Sound because having a AudioSource object allows you to group sounds together and then set the position of the AudioSource instead of setting the position of multiple ig.Sound instances.

[collinhover]

Hmm, so the AudioListener is decoupled from the sound. Does this mean multiple AudioListeners can listen to the same sound then? I'm not convinced about the AudioListener, but I do like the idea of the AudioSource. I'd like to see the code for both! Can you make a pull request?

[arakash92]

Sure. coming up!

[arakash92]

Actually there seems to be a bug I need to fix first.

[arakash92]

Hm. I'm gonna submit a pull request if you wanna take a look at it. There is a strange bug though I'm not sure what it is yet.

[arakash92]

Forgot to reference this issue in the commit but I have submitted a pull request :P

[bdaenen]

I've indeed been thinking this out and trying to get the web audio API to work, which is turning out to be quite the pain :D of course as soon as one reads what amazing things it's capable of (cross fading, all sorts of filters, positioning, ...) one gets distracted from what the original goal was :)

Either way, I want to extend ig.Sound with a sourceEntity to handle positioning. The idea is to use the Web Audio API when available to the browser (Firefox 25+, safari 6+, opera 16+ and chrome since ages) and only use the extra functionality when possible to keep compatibility.

As for the listener, I honestly only see limited use in having multiple listeners. One of the only possibilities I can think of is a stealth (portion of a) game, where guards can hear sounds only when they're within audible range. This is doable in simpler ways, but using actual sound is probably the most accurate one as volumes can differ throughout the duration of the sound.

Another question is if we should make the camera the listener, or the entity the camera follows. Following the entity would mean the sound would also change depending on what direction the player is running in, but I'm not entirely convinced that's an improvement (turning left/right would make sounds switch from one speaker to another, which sounds irritating for headphone users :))

In a later stage I'd also like to extend ig.Music. This would be useful for things like perfectly looping music, cross fading into other tracks (and for instance making music more/less intense whether you're in combat or not by cross fading to a different version of the current track), adding filters to simulate the current environment, ...

I'll be looking into getting the API working in Impact today, rather than just fiddling around with it :) if I get anything done I'll get back to you.

[collinhover]

I'm starting to come around to the idea of AudioListeners, but they should probably extend ig.EntityTrigger so they have all the basic entity properties such as distanceTo, moveTo, and tweening methods, as well as the possiblity of being added in Weltmeister (maybe), etc. Extending the trigger also allows them to be used in the trigger chain, where a player can collide with and activate a listener, or another trigger can activate a listener, etc.

In terms of the AudioSource, the larger issue is that ImpactJS has no parent/child system to create groups of entities. This is essentially what the AudioSource does for sounds, and as a short term fix it is okay. Honestly, I don't really know if I want to bother with adding parent/child transforms at this point. On a side note, we shouldn't be running a for... in loop in the AudioSource update. AudioSource should maybe extend ig.EntityExtended and only do it when changed === true (i.e. it was moved). If we do this, then AudioSource should have performance of movable and do the volume update in updateChanges after calling parent() to update the current moveTo.

[arakash92]

Awesome! You seem to have a better grasp of this than me. I'm looking forward to hear more 👍

[bdaenen]

Got the basics for positional audio working :) just need to configure it better as right now it seems to reduce the sound a bit too much! Turned out to be really easy with the web audio api, as there's a built-in "panner" to pretty much do the job for us :D

[arakash92]

whoho ;D can't wait to try it!

[bdaenen]

I've cleaned up some code today and worked out some better settings (by default the volume was reduced by far too much) and I seem to have found some stable point for my tilesizes :). I'll add some configs for easy configuration. Things that can be changed are (bold = default):

• the distance model (linear, inversed, exponential)
• the panning model (equalpower, HRTF)
• the reference distance from which to start reducing the volume (1)
• the maximum distance to reduce the volume up to (10000)
• the rolloff factor (how much the sound is reduced in volume per distance "unit", default = 1)
• the cone inner angle, an angle at which the volume should not be reduced (360 by default = no reduction other than normal distance reduction)
• the cone outer angle, outside of this angle the volume will be reduced (inverse of inner angle. 0 by default = no reduction other than normal distance reduction)
• the cone outer gain, which is how much the volume should be reduced outside of the outer angle (0 by default = no sound can be heard in the outer side of the cone)

It's falling back to the default API quite gracefully and performance seems fine. The only thing slightly bothering me is that I can't get it to play equally loud from both speakers when being close to the source. I guess in the worst case we can make a config for adding an area where the pannerNode becomes disconnected so sound will be played the normal way.

In a later stage I'd also like to optionally add a Doppler effect for things with a velocity, and let it rotate with it's source entity (which is required to make proper use of the cone attributes). I'll also add support for manually adding extra filters etc and try to write up some documentation to get people on the right track.

For more info you can check:

• pannerNode article on MDN: https://developer.mozilla.org/en-US/docs/Web/API/PannerNode
• the web audio api pannerNode spec: http://www.w3.org/TR/webaudio/#PannerNode-section

[arakash92]

awesome!

[collinhover]

Nice work @bdaenen. This combined with lighting and camera atmosphere is going to allow for some really interesting effects.

On the issue with getting sound to play equally when close to a sound, perhaps there a modifier that starts at 0 at a certain radius from the listener, and lerps to 1 at another smaller radius, which can force equalization?

[bdaenen]

@racingcow: Other than sprite-audio I don't see too much of an advantage in using howler.js :) many features are already present in vanilla impact as is (detection/selection of compatible audio formats, individual/global volume adjustments, multichannel, pause/play/stop, fading, ...) and others were quite easy to handle (fallback to standard html5 audio).

I'd also like to allow developers to add their own nodes and filters to sounds, so that if they want more than just positioning and some built-in filters they can dive into the specs and still access everything through the actual API and easily add them in whatever order they want :)
Currently everything audio-related can be used exactly the same as before so we're not breaking compatibility with anything whatsoever, even if you're coming from vanilla impact.

@collinhover I'll probably be done with the positioning part including fallback and configs somewhere early this weekend. Would it be okay if I added functionality like manually adding filters (with some standard common filters added perhaps? I'm thinking about stuff like a telephone filter for instance), extended music with crossfading, ... in later pull-requests :)? That way I can also easily adapt to structural changes without having to rewrite everything, and we'll get some stuff in faster.

[collinhover]

@bdaenen yep, lets start with solid basics before we worry about the extra features. Always best to make sure the core works really damn well. Also, have you tested http://jessefreeman.com/dev-diary/optimizing-super-resident-raver/ at all while working on sounds?

[bdaenen]

@collinhover Yup, I've been looking into this but the issue with the suggested "fix" is that it breaks multichannel sounds entirely :) a sound won't be able to play twice at the same time and will simply stop, rewind, and play again from the start when trying to play it the second time which just sounds bad.

I'm having some issues finding a better fix for this though. Currently whenever Impact loads a sound, it checks if it's multichannel (default = true) or not. If it is, it will check the amount of channels impact is currently using (ig.Sound.channels) and create a new Audio object for each channel, which unfortunately results in multiple requests in almost all browsers, even though the src is the same... (Opera doesn't seem to, so it is possible). To me this looks like a shortcoming in browser caching.

@phoboslab wrote an article about this while he was working on Impact and figured out some solution back then, but unfortunately after some tests I've found out it currently only seems to fix the issue for firefox. Here's a testpage so you can see for yourself :)

[collinhover]

Interesting. Is there a way with the web audio API to convert a sound file into a native audio, or is that something that takes way too long?

[collinhover]

Can we force single channel sound and then defer the multichannel extra loading until after the game has started? Also, it might be good to just force single channel sounds always for mobile to keep data use down.

[bdaenen]

If there's a way to check for mobile, we surely can :) I'm thinking instead of adding them immediately we can bind an event on the initial one's load and then clone them :)? As long as all browsers fire it we should be safe.

[collinhover]

I think impact does user agent sniffing that includes mobile. Try ig.ua.

[bdaenen]

Fixed most of the multiple loading issues (except in Chrome, it doesn't seem to care) and am working out some other browser inconsistencies and bugs at the moment. Had to rewrite quite a lot to get everything working nicely in firefox, but I'm getting there :)

[collinhover]

Can't wait to try it. How many channels/sounds can be running simultaneously?

[bdaenen]

Hard to really test :) I'll look into it later as I currently don't have any time, but most browsers give up in the loading phase, as it's quite the memory hog (each channel/sound gets loaded into the memory separately, without preloading things can quickly get very very bad)

[bdaenen]

@collinhover said memory issues could possibly be solved by creating more channels on demand? I'll look into the possibility of dynamically generating more channels (within a min/max limit which we could use configs for) but I think it'd turn out okay as long as it's created as soon as we used up our last channel. This could potentially greatly reduce memory! usage for sound effects that aren't likely to play more than once at any given time.

I'll also check performance differences between creating a mediaSource during load and storing it in memory vs creating it whenever we try to play a sound. Currently I'm creating a source of every Audio as soon as all channels are available, but memory-wise this seems quite inefficient.

Edit: The dynamical loading works like a charm in all browsers... except for Chrome, which seems hell-bent on downloading the file every time again and again, resulting in a delay before it plays :( I suppose we could somewhat counter this with always keeping 1-2 channels ahead.

Performance-wise, browsers generally tend to make audio glitch or crash way before I notice any slowdowns :D Chrome managed to handle about 100 before glitching became very apparent, but at the 150 mark you couldn't even hear anything proper anymore. Firefox Nightly gives up and crashes entirely around ~115 :) I tested this with files from about 1.30 mins so i was able to periodically add more while keeping track of the amount and not having the first one finish before I was done. In all cases I didn't notice any slowdown whatsoever.

another edit: Sorry for the delays, I've been rewriting most of the thing to improve performance, fix some bugs and add some features. Right now the only thing I still want to fix before sending you a pull request is implementing some sort of "deadzone" where the panners gets disconnected to play the audio unreduced through both speakers for whenever the player is near it.

Loading extra channels is now done dynamically. There's a config for the minimum and maximum amount of channels to load and one for the amount of channels to buffer. These "buffer" channels will only be of any use for browsers which are determined to keep downloading the same file over and over (chrome/safari) as it seemed the only way to prevent pauses in between of triggering "play" and actually getting output.

Sorry for the long posts :)

[collinhover]

I didn't realize each sound took up so much memory. Are they streaming our just fully loaded?

[bdaenen]

Impact creates an Audio object for ever channel of every sound when loading up the game and puts it into the soundmanager's clips property. The Audio objects automatically pre load their content so they do stream, in a sense, but they all start pre loading from the very start regardless if you need/play them or not.

Edit: Just to clear things up, the crashing of the browser and glitching of the audio is not due to memory constraints, but rather the API's limitations. Playing > 50 sounds at the same time is a very, very unlikely situation anyway, let alone > 100 :). That doesn't mean this is ideal though.

When we come up with a viable way for handling deferred loading properly all of this shouldn't form any issues anymore, though, although it'd properly require massive refactoring of pretty much everything asset-based. After the amazing work you've done on getting top-down working I don't have any doubts though :D

[collinhover]

Ah, I've been reading your replies via email and missed your edits. This is sounding great. Because some browsers tend to glitch and crash at a certain number of audio objects, do you think we should cap that conservatively and allow the dev to config it to a higher value? For example, I would imagine 10-15 sounds is about all that would be needed before it becomes a mess, but what is your testing suggesting?

[bdaenen]

@collinhover Just loading audio tags doesn't cause any issues, it's only when we try to play > 50-75 simultaneously that the glitching starts to kick in :) I'm not sure if it makes a lot of sense implementing a limit on this as it seems very situational to have this many sounds playing at the same time. It's also very uncomfortable to listen to such an amount of sound effects at the same time, believe me :D

[collinhover]

Okay, that makes sense!

[bdaenen]

My apologies for disappearing for a while, things got busy and some personal stuff came up. I've been fixing, refactoring and testing stuff in different situations, and have added some example sound effects to the impact++ run & jump demo.

One more question: How do we handle dieing entities? Currently I'm using sourceEntity.onRemoved as a signal to remove and clean up the sound object, but when objects "die" offscreen they don't go through their death animation, and just instantly get removed, which results in no sound playing at all. I can imagine we'd like some death sound effects on many entities but I don't see a proper way to do it currently. Waiting until all the current channels for this object have ended playing also sounds flawed to me, but I'm not sure how else we could handle it.

Any good ideas :)?

[collinhover]

Can you give an example situation where you'd want to hear sound for an entity dieing off screen?

We could add a property to the core entity that allows devs to override the instant death when off screen, in case they want to play the death sound in all cases. Would this do what you are suggesting?

[bdaenen]

An example would be the player throwing a grenade. I figured it made sense for the grenade entity to hold the explosion sound effect (allowing different projectiles to have different sounds, regardless of which ability fires them), but whenever they drop out of the screen they'll silently die.

Overriding the instant death sounds good! It probably wouldn't be too big of a performance hit either as offscreen drawing is much faster than onscreen drawing :)

[collinhover]

Entities never draw when off screen, so we don't need to worry about that in any case. Internally I've added a canDieInstantly property to ig.EntityExtended, which defaults to true. You can set it to false to disable instant death when off screen, but in either case an entity will always die instantly if it has no death animation. I've also split the kill method into kill and death, where death is called by kill as long as the entity is not dieing silently. I'll push these updates soon.

Also, if you're using the onRemoved signal, that is dispatched during the entity's cleanup method, which is called just before the entity is removed from the game. In other words, when onRemoved is dispatched, the entity has already finished its death animation. I'd recommend creating the sound effect by overriding the entity's death method (again once I push the change), and have the sound clean itself up once it is done playing. I think having a way for sounds to play once and remove themselves would be a very good option. Can they do this now?

[bdaenen]

The onRemoved is fine for cleaning up the sound automatically I think, its simply used to detect removal of the source entity (if there is one) to trigger it's own destroy. No playing is done here :)

For playing a sound effect just once I can either add another option to the constructor or add a playOnce method. The first option sounds the easiest. All that we'd need to do is prevent binding the destroy on entity removal, and bind an event on the used channel's onended to call destroy :)

[collinhover]

Sounds good. Perhaps name the play once and destroy property suicidal? This matches the same property on the trigger entity. On the trigger the suicidal property is true by default, do you think this would be good for sounds as well? You've probably done this, but just in case you haven't, it'd be good to try to match some of the method/property names from ig.Animation, such as playFromStart( once, reverse ).

[bdaenen]

suicidal sounds good :) as for playFromStart, impact plays sounds from the start by default whenever play is called. I'll add an optional time argument to play so a sound effect can be played from a certain point in time and add a playFromStart method for consistency :) reversing should be quite easy as well if all browsers support it. I'll have a look at the animation methods and adjust / add where required :)

[collinhover]

@bdaenen don't worry about missing r6, whenever you're ready with the audio we'll merge it into r6 and re-release it, or just have r7 be the audio specific release. I went ahead with r6 because I wanted to wrap up all the recent changes.

[bdaenen]

Sounds good :) I'm currently breaking my head over all the possible use cases, some more browser inconsistencies (read as: working around the various bugs still present in various browsers) and some refactoring. From the point where I thought I nearly had it all down(about a month ago) I pretty much rewrote every line of added code, and that's still only for sound effects :)

[bdaenen]

I finally seemed to have worked out most quirks I was having in Firefox and simplified a couple of things :) if I don't run into any really big issues anymore this should (finally) be properly usable soon. My apologies for the amazingly long delay :(

I'll be testing fallback to default sound a bit more today along with some more situational cases and should be able to shoot you a pull request relatively soon (after all this I don't dare to promise anything specific anymore :D). I expect you'll want some explanation here and there, and most likelysome refactoring for the better, but hopefully that won't make any new issues surface :)

[collinhover]

Sounds great, looking forward to it!