Implement / bind non-blocking API (portaudio callback)

[jeremyletang]

Portaudio provide a non-blocking API using callback function which is called each time some new data are coming. For now this functionality is not implemented in rust-portaudio as a user can do the same thing using a rust task and calling the blocking api inside of it.
Maybe we can bind this functionality too as the native implementation is probably more effective.

[smosher]

This is a blocker for using Rust with PortAudio for me.

I've bumped into a few problems with the blocking API (in C and Rust) and in some cases merely copying from one device's buffer to another would under/overrun terribly. (Latency is admittedly higher than it should be in that configuration, but with the callback API the under/overruns can be pushed down to virtually nothing.)

[mitchmindtree]

@smosher any chance you could elaborate a little more on the problems you've run into? I'm unsure how changing from the blocking callback to a non-blocking would affect copying from one device's buffer to another? (Sorry, I just haven't had a great deal of experience with PA is all!)

@jeremyletang do you have any ideas on how we'd expose this to users of rust-portaudio? I actually agree with your point in #20 that it would probably be much more elegant to use Rust's tasks (soon to be named threads) than to try and bind the C non-blocking version, though perhaps there's some benefit to doing this that I don't yet understand! I'll have to get around to reading the PA docs to get some clarity, but right now I'm just assuming that at its core the "non-blocking" C api is just spinning up the "blocking" callback on a separate thread?

Just for the record, I've been using the "blocking API" in a non-blocking way by spawn(proc(){...})ing the instantiation of the portaudio stream and communicating with it via Rust's channels. I've found this solution to be elegant, fast and super safe, as the channels act like FIFOs allowing for nice, lockless communication between the audio thread and other threads. I've been porting across my synthesis engine and using rust-dsp as a dsp-chain and I've been able to do some fairly heavy synthesis stuff (300+ simultaneous oscillators driven by interpolating between amp and freq envelopes) without glitches or overflows. Hopefully I can add it as an example to rust-dsp once I untangle it from my generative music engine!

[jeremyletang]

hi,

@smosher sorry i've just not see your answer.

@mitchmindtree I think we can add the C version of the callback, but encourages user to make their own Rust Callback using Rust threads.

I have some idea yes to implement C callback.

We can add a function to the api like :

pub fn open_callback<F>(&mut self,
                        input_parameters: StreamParameters,
                        output_parameters: StreamParameters,
                        sample_rate: f64,
                        frames_per_buffer: u32,
                        stream_flags: StreamFlags,
                        callback: F) 
                        -> Result<(), Error>
                        where F: FnMut<(S,), S>

where S is the type parameter of Stream.

So the user can pass it's callback as a function / closure / object wight implement FnMut<(S,), S>, and we can use the user_data parameter of the Portaudio callback to pass a rust-portaudio internal struct, which can call the callback.

[mitchmindtree]

@jeremyletang ahhh cool, that makes sense! Looks good to me 👍

[smosher]

@mitchmindtree @jeremyletang sorry for sleeping on this.

Honestly, I haven't figured out what is causing the problem with the blocking method, but the trouble it ran into was underrunning or overrunning the buffer. I thought the automatic management could at least take care of that, but no such luck. FWIW, I'm just dumping the output of a USB turntable's audio device into the onboard sound. With the callback API, if I run into an over/underrun, I just adjust the read or write pointer to give it some headroom but that is rarely invoked (minutes to hours apart.) It's possible PortAudio simply can't measure the latency correctly.

I am not too concerned with this case, but when I go to write a sampler I would very much like to run the mixing and effects processing on demand. I've done something similar with PortAudio in the past and I had very low latency. I'm not sure what that should look like with the blocking API.

I am blocked on some other things, so this isn't in any way urgent to me.

@jeremyletang The solution you presented looks good to me too.

Spitballing: I wonder if an iterator interface would perform well. All the audio streams really want are to pull some bytes in sequence. Writing iterators might be trickier than writing your own callbacks, but I think reusability could benefit.

[mitchmindtree]

@smosher I like the idea of using an iterator! I've been planning to do this for rust-dsp, I quite like the design of Piston's event-loop and was hoping to do something similar but for audio instead.

Perhaps we could define a few events like this:

pub enum Event<I, O> {
    Update(DeltaTime),
    In(AudioBuffer<I>),
    Out(&mut AudioBuffer<O>),
}

Portaudio can drive the loop under the hood, and usage could end up looking something like this?:

for event in events {        
    match event {            
        Event::Update(dt) => { /* check channels for msgs, general updatey stuff */ }
        Event::In(incoming) => { /* use the incoming buffer here */ }
        Event::Out(outgoing) => { /* fill the outgoing buffer here */ }
    }
}

These are missing heaps of stuff but I'm sure you get the gist. Thoughts?

@jeremyletang do you think it's best if we leave rust-portaudio as a nice interface for the bindings and do this in something more high-level like rust-dsp? or would you be happy to see this implemented here?

I was just thinking that perhaps we could start a "sound_stream" repo (maybe under the PistonDevelopers umbrella?) that depends on rust-portaudio and do it there? Maybe a new audio-focused umbrella altogether could be a good idea (i'll post a separate issue).

[smosher]

@mitchmindtree I think you're right... it's probably best to match PortAudio functionality in this library, then build something else on top of that.

Events are probably fine... but keep in mind you can have multiple inputs and outputs, and which buffer is ready might make a difference as to what you intend to do with it. I had a different design in mind (still iterator based, but connecting endpoints directly.) The differences are probably a matter of taste, but feel free to ping me in the appropriate issue if you want to hear about it.

[mitchmindtree]

Ahh yep, so I guess we could have a DeviceId or an enum? i.e.

pub enum Event<I, O> {
    Update(DeltaTime),
    In(DeviceId, AudioBuffer<I>),
    Out(DeviceId, &mut AudioBuffer<O>),
}

Or something along those lines anyways. I guess I'm only really throwing around basic ideas. I'd definitely be keen to see what sort of design you had in mind :) I might fire up another repo shortly and ping you both.

[mitchmindtree]

@smosher @jeremyletang I've started a thread on the topic here.

[jeremyletang]

@mitchmindtree, I think this library should stay nears to the C version, but in a rusty way. An organisation to aggregate all audio stuff in Rust (higher level api, audio format, etc..) will be really cool !

[safiire]

Is this project still active? Is there some problem with exposing the callback in Rust?

[mitchmindtree]

@safiire Yep! I think it's not implemented yet just because no one has had a chance to do it yet or needed it enough - most people who need a non-blocking audio thread seem to just spawn a separate thread and run the blocking instance on that. Would be nice to have this implemented at some point though, shouldn't be that tricky at all.

[safiire]

Ahh, cool, ok thanks for the response :)

[killkrt]

It would be great having callback mechanism. Blocking could cause a lot of unwanted latency. Is there any estimation about when it will be implemented? I mean it's something that it will happen by weeks, months or maybe?

Thanks for your great job!

[mitchmindtree]

@jeremyletang @smosher @safiire @killkrt done! See #68 for the PR, or check out the example.

[safiire]

Woo, thanks :)