Fix choppy playback when calling ofSoundPlayer.setPosition (in develop)

[edburton]

I'm working on an iOS app that uses ofSoundPlayer.setPosition to jump to different times while playing back a sample. In OF 0.7.4 it works great with seamless playback while jumping back and forth in one sample. I'm a little worried as when testing the app against the develop branch of OF the playback is no longer seamless, calling ofSoundPlayer.setPosition becomes noticeably choppy.

(p.s. GitHub apology, I couldn't work out how to give this issue an iOS label or move it to be "(in develop)", am I missing something?)

[bilderbuchi]

no apologies needed, you don't have the permissions to do this. :-)

[admsyn]

Hi @edburton, I'll look into this. The difference between 0.7.4 and 0.8.0 is that we've moved from using the ofxOpenALSoundPlayer to a new sound player (ofxiOSSoundPlayer) which uses AVFoundation (instead of OpenAL).

If you need a quick fix, you should be able to get the same behaviour as before by grabbing the ofxOpenALSoundPlayer from 0.7.4.

[admsyn]

Can I get some details of your setup? Stuff like:

• File format that you're using
• Size of the audio file
• Estimate of how long the cut-out is
• How much you're jumping around
• iOS version

EDIT: and if you're using MP3, can you tell me if you still get the same issue with WAV or AAC?

[edburton]

• File format: .caf (little endian 16 bit, 44,100 sample rate)
• File size: 50 MB
• Cutout: Less than 1/10 second maybe?
• Jumping around: About twice a second
• iOS version: 6.1.4

[admsyn]

Alright, I haven't had much time to look into this but it's semi-urgent due to the impending 0.8.0 and the fact that it's a pretty legitimate regression.

@julapy?

I'm not sure how much can be done, since the AVAudioPlayer API is pretty fire-and-forget. I don't think there's many places to force the player to optimize for things like instant seeking. As far as I know, the only real "optimization" work that can be done in AVAudioPlayer is just scheduling future playback to happen at an exact moment, which doesn't cover this use case.

Might be worth investigating if WAVs and AACs have this issue.

[ofTheo]

@edburton also might be worth trying an mp3 or m4a - weirdly the AVAudioPlayer can be fast/faster with those formats.

the OpenAL based player loaded the whole sound into memory ( uncompressed ), so that could explain the fast seek times.

@admsyn do you think there is a way we can add an option to decode the sound buffer into memory?

[admsyn]

Not with AVFoundation, as far as I know. IIRC the design of many AVFoundation bits intentionally tries to separate you from the raw stuff like buffers and decoding so it can intelligently make use of onboard decoder chips and such (grain of salt).

The AVFoundation player has many improvements over the old one, but this is a specific use case where the old system really did work best. I'm not really sure what'd be the best way to handle this, since this really is like a 1% use case, where we want to do sampler / MPC style instant seeking (I assume?).

FWIW I think dipping down to the Audio Unit level is the way to handle the tight playback control you'd need to pull this off. The iOS OpenAL implementation kind of hung out in this nether region that was basically a thin wrapper over straight Audio Unit / Core Audio stuff.

.. thinks ..

[admsyn]

I have an implementation of an oF-friendly wrapper over the AUFilePlayer (an audio unit that's available on OSX and iOS). This is geared towards OSX and the ofxAudioUnit "architechture", but it might be something we could tweak for iOS super-fast to get it ready for 0.8.0 in the case that: A) We don't find a workaround, B) this regression is unacceptable and C) bringing back the ofxOpenALSoundPlayer would involve too much pain :)

https://github.com/admsyn/ofxAudioUnit/blob/master/src/ofxAudioUnitFilePlayer.cpp

EDIT: just to clarify, I'm saying this could be provided as a secondary / fallback sound player in the case where you want to do things like @edburton

[edburton]

I've just tried WAV and AAC, no change.

Do you think it'd work if I have two ofSoundPlayer's and faded from one to the other? I should be able to work out in advance where I need the next playhead jump to be, so there might be some sort of work-around where I call setPosition a few moments before play. Trouble is I need to know that the timing is going to be precise enough to keep percussion in sync as I jump/fade from one beat to another.

I must admit; I did think I was a surprisingly lucky 1% when the old version worked so seamlessly.

[admsyn]

We could try calling prepareToPlay in the init process. I don't think it'll have much effect outside of the initial "play" call, but it's worth a shot.

@edburton drop this line:

[(AVSoundPlayer *)soundPlayer prepareToPlay];

On line 26 of ofxIOSSoundPlayer.mm (after the loading stuff is done, before return bOk;)

[edburton]

@admsyn when I try that I get a warning...

"/Users/ed/Documents/openFrameworksGithub/addons/ofxiOS/src/sound/ofxiOSSoundPlayer.mm:26:6: 'AVSoundPlayer' may not respond to 'prepareToPlay'"

...and when running it throws...

2013-07-25 17:56:17.259 Tunetrace[2098:907] -[AVSoundPlayer prepareToPlay]: unrecognized selector sent to instance 0x1962fc0
2013-07-25 17:56:17.263 Tunetrace[2098:907] *** Terminating app due to uncaught exception 'NSInvalidArgumentException', reason: '-[AVSoundPlayer prepareToPlay]: unrecognized selector sent to instance 0x1962fc0'

[admsyn]

Ah, I was looking at the wrong abstraction layer :) prepareToPlay is already being called in the AVSoundPlayer.

[admsyn]

@edburton RE: your previous comment about keeping two sound players. That would work, because the underlying AVAudioPlayer has an affordance for scheduling future playback quite reliably. I don't think that's exposed in the ofxiOSSoundPlayer wrapper, though.

If it were to be exposed in the wrapper, it'd be a function you call like:

soundPlayer.playAtTime(whenToPlay);

where whenToPlay is a float representing a number of seconds from now (so, calling it with 0.5 would be "play half a second from now"). Would that work for your app?

[edburton]

soundPlayer.playAtTime(whenToPlay) might do very nicely. Is there any chance it could come with a matching soundPlayer.stopAtTime(whenToStop) perhaps?

[julapy]

@admsyn @edburton,
think you can load a sound, pause it, jump to position, and play when you are ready.
thats what you're describing above right?

@edburton, think the easiest solution in your case would be to jump back to using ofxOpenALSoundPlayer.
the API should be exactly the same as they both extend ofBaseSoundPlayer.

[edburton]

@admsyn I think I probably am too eccentric an edge case to be drifting from the ofBaseSoundPlayer API when you're all so close to 0.8. I promise not to sulk when 0.8 comes out if my lazy jumps become slightly choppy.

@julapy yep, pause, jump, play, repeat, with two playheads, is what I had in mind. It would have the advantage that I'd be able to do a slight fade rather than a cut. I am in a hurry to deliver a commercial project though so if I can stick with ofxOpenALSoundPlayer I might do that.

Two remaining questions then. A) Do people expect ofxOpenALSoundPlayer to carry on working with iOS7? B) What would be the tidiest way to use ofxOpenALSoundPlayer while still being able to pull from the develop branch as 0.8 approaches?

[bilderbuchi]

What would be the tidiest way to use ofxOpenALSoundPlayer while still being able to pull from the develop branch as 0.8 approaches?

hm, [cs]hould we farm it out into a mini-addon, for those 1% cases?

[ofTheo]

@edburton - best is to just copy the old files into a new folder in the addons folder.
then add those files to your app project and just use #include "ofxOpenALSoundPlayer.h"

The reason we removed it, is it is really buggy in terms of deallocation ( lots of crashes on exit issues ) and the c based code is quite a nightmare to work with.

I would be interested in getting better seektime performance for the new sound player though.

some relevant SO pages:
http://stackoverflow.com/questions/2699093/avaudioplayer-seek-performance-is-not-good
http://stackoverflow.com/questions/882753/avaudioplayer-currenttime-problem

[edburton]

Oh my, I would be a very appreciative 1% if that were possible @bilderbuchi , as well as my own http://tunetrace.com it'll be a couple of large brands so it'll be helpful towards overall plan of OpenFrameworks global domination :-)

ah, thanks @ofTheo, that is really useful. I've been noticing those crashes on exit and am somewhat relived to be able to blame ofxOpenALSoundPlayer rather than my probably even more suspect OpenCV hacks :-)

[admsyn]

I tried the experimenting with the

[audioPlayer stop];
[audioPlayer setCurrentTime:position];
[audioPlayer prepareToPlay];

trick documented in the second SO link @ofTheo posted, without any noticeable difference.

One thing I'm currently wondering: Is the AVAudioPlayer doing a quick fade out / in every time the audio changes positions to avoid clicks and pops? It sounds very "smooth" to me whenever the audio jumps around, where I'd expect there to be a bit of pop if it were actually cutting out then starting up again. FWIW I've seen people run into issues like this with AVAssetReader, where it was ramping up the first ~30 ms of any audio that was read in (grain of salt, again).

If so, that'd be some 💩

[admsyn]

So here's what the seeking looks like:

It looks like there's a bit of a fade in / out being applied, but it's also taking a while to buffer up the sound. I'm pretty sure this isn't solvable with the AVAudioPlayer-backed sound player.

[ofTheo]

closed by #2363

@edburton @admsyn added the OpenALSoundPlayer back into the iOS project - so you should be able to use that instead of ofSoundPlayer