AudioMixer with effects

[JohnStabler]

It would be nice if effects could be added to the voices on the audio mixer. For example: pitch (playback rate and/or time-stretch), reverb and EQ maybe?

This can't be done near-realtime in Python, so it is better suited to a compiled library. Most effects would require buffering but should be achievable on something like a Pico.

[gamblor21]

I wanted to add effects on synth channels and saw this issue also about adding audio effects. After some discussion on discord and looking through the code and some other audio libraries I see all the audio code uses get ..._get_buffer_structure and ..._get_buffer to pass information from one to another. In this way audio components can be chained together. E.g. synthio.Synthesizer can play into audiomixer.Mixer which can play into audiobusio.I2SOut to get to the speaker.

My idea is to create a new module AudioEffects then it is easy to exclude for smaller builds. The effects would each implement the buffer protocol in the same way current audio modules do. Hooks would have to be built into any audio module that wants to support to the effects. Something like mixer.voice[0].addeffect(effectName). Assuming a common protocol then a similar .addeffect can also be added to a synth or synth channel so the developer can add the effect at the point they want. In theory effects could be hooked between each other.

At this point I have done a proof-of-concept to see if a basic echo was possible on an RP2350. It worked fine and the memory allowed for a long buffer delay. The POC was done without setting up a module just an in-place test. If this sounds like a decent idea I'd next start on a more complete test with a separate module and maybe hooked into two+ audio components.

Anyone have any thoughts or comments?

[tannewt]

I wouldn't do mixer.voice[0].addeffect(effectName). Instead, I think the effect could be a separate object that you'd "play" into the mixer voice or vice versa.

I like the idea of separate modules for different effects.

@todbot and @jepler may have thoughts too.

[gamblor21]

Putting some ideas down both for others to critique and to organize my own thoughts:

This is basic code to play a note:

audio = audiobusio.I2SOut(...)
mixer = audiomixer.Mixer(voice_count=1, channel_count=1,...)
audio.play(mixer)
synth = synthio.Synthesizer(channel_count=1, sample_rate=44100)
mixer.voice[0].play(synth)
note = synthio.Note(261)
synth.play(note)

There are two potential ways to add effects. The first as tannewt suggested:

# ... similar to above
echo = audioeffects.EffectEcho(delay=50, decay=0.7) # define an echo effect
echo.play(synth)
mixer.voice[0].play(echo)

In theory effects could be chained together e.g. echo.play(chorus.play(synth))
Pro for this method is that almost all existing audio objects would support effects with no code change.
The downside is I'm not sure how easy to would be to dynamically change effects on the fly and the effects are only granular to an audio object like Mixer, WaveFile, Synth and not to a specific synth channel. Potentially a special case could be added for synth channels.

Another way is to add effects to audio objects.

# ... similar to above
echo = audioeffects.EffectEcho(delay=50, decay=0.7) # define an echo effect
mixer.voice[0].play(synth)
synth.addeffect(echo)

This method also allowed effects to be chained, call addeffect again e.g. synth.addeffect(chorus)
Pros for this method include dynamically changing effects, and likely being able to add effects on a per channel basis in synth.
The downside is having to add addeffect to any audio object that wants to support effects.

Anyone have any thoughts? Are dynamically changing effects a good idea? Would we want effects on a per channel basis in the synth?

In the meantime I'm going to keep trying things out to see what may work best as time permits.

[todbot]

First, this is very cool!
I like the .addeffect() technique a bit more, for the reasons @gamblor21 mentions, but I'd rather have it add to Mixer instead of Synthesizer, so that other items played with the mixer can get the benefit of the effects. (Many physical mixers do this, having effects busses you feed channels into)

So that would I guess look like:

echo = audioeffects.EffectEcho(delay=50, decay=0.7) # define an echo effect
mixer.voice[0].play(synth)
mixer.voice[1].play(wavfile)
mixer.addeffect(echo)  # add effect to mixed output
# and then also
echo2 = audioeffects.EffectEcho(delay=150, decay=0.7) # define another echo effect
mixer.voice[0].addeffect(echo2)  # add effect only to voice 0

[tannewt]

Anyone have any thoughts? Are dynamically changing effects a good idea? Would we want effects on a per channel basis in the synth?

I think it's important to think about how you'd remove an effect as well. I do think we want to change them dynamically.

I don't think we need them on a synth channel basis. Instead, you can have two synths.

[todbot]

I don't think we need them on a synth channel basis. Instead, you can have two synths.

So no audio effects for audiocore.WaveFile and audiocore.RawSample?

[tannewt]

I don't think we need them on a synth channel basis. Instead, you can have two synths.

So no audio effects for audiocore.WaveFile and audiocore.RawSample?

That's not what I meant. I was responding to "Would we want effects on a per channel basis in the synth?".

I still think the best way is through separate intermediate objects that get played.

echo = audioeffects.EffectEcho(delay=50, decay=0.7) # define an echo effect
echo.play(synth)
# with echo
mixer.voice[0].play(echo)
time.sleep(1)
# without
mixer.voice[0].play(synth)

[gamblor21]

After playing around with the code all weekend the first way @tannewt suggested I think makes the most sense. Looking at how the Teensy audio library works it seems similar.

# give the synth and echo effect
echo = audioeffects.EffectEcho(delay=50, decay=0.7) # define an echo effect
echo.play(synth)

# give a wave file a chorus effect
wavesound = audiocore.WaveFile("wave.wav")
chorus = audioeffects.Chorus(voices=5)
chorus.play(wavesound)

# combine them in a mixer
mixer.voice[0].play(echo)
mixer.voice[1].play(chorus)

# add a reverb to the overall mixed sound
reverb = audioeffects.EffectReverb()
reverb.play(mixer)

I have to look closer but in some instances playing objects into others may reset buffers / reset the sound but that can be tweaked easily enough.

I am getting closer to having some code I can push as a draft PR to give something more concrete to look at.

[todbot]

reverb = audioeffects.EffectReverb()
reverb.play(mixer)

Normally mixer is fed into the actual audio player, providing the much-needed audio buffer to the whole system, so I'm not sure how the above would work. Is this a typo?

Would representing the signal chain in code, rather than as a data structure, create glitches when the chain is changed? Let me think through this with an example. e.g. let's set up a synth playing through chorus & reverb, then remove the chorus:

# standard audio setup
audio = audiobusio.I2SOut(...)
mixer = audiomixer.Mixer(..., buffer_size=2048)
audio.play(mixer)
synth = synthio.Synthesizer() 
# mixer.voice[0].play(synth) is what we'd normally do here, instead...

# wire up effects: synth -> chorus -> reverb -> mixer -> i2s
chorus = audioeffects.Chorus(voices=5)
chorus.play(synth)
reverb = audioeffects.Reverb()
reverb.play(chorus)
mixer.voice[0].play(reverb)

# time passes, remove chorus from signal chain: synth -> reverb -> mixer -> i2s
reverb.play(synth)   # would this cause a glitch? 

I forget if doing the equivalent does in the TAL. I'll see if I can find my Teensy audio boards and try it. If all audio effects have a "wet/dry" mix knob and zero-effort pass-through case when dry=100%, then we can get glitchless removal of an effect without altering the signal chain.

[gamblor21]

reverb = audioeffects.EffectReverb()
reverb.play(mixer)

Normally mixer is fed into the actual audio player, providing the much-needed audio buffer to the whole system, so I'm not sure how the above would work. Is this a typo?

This was more an example of "you could do this", then any practical or good idea.

Would representing the signal chain in code, rather than as a data structure, create glitches when the chain is changed? Let me think through this with an example. e.g. let's set up a synth playing through chorus & reverb, then remove the chorus:
reverb.play(synth) # would this cause a glitch?

I'm still not sure about this case. I do want it to work, or to have a way to switch effects in/out at runtime. I'm just not sure the optimal way to do that yet.

I did get a "do nothing"/pass-thru effect working tonight. Next step to make it do something.

[relic-se]

+1 on @tannewt 's suggestion of running audio buffer sources through the effect and then to the final mixer object before the output. In a way, I think it is reminiscent of guitar pedals as to how you'd chain them together. Though I know the naming scheme we're playing around with isn't final, I think audioeffects.EffectReverb is somewhat redundant and should be more like audioeffects.Reverb. There could be some minor conflict issues when using from audioeffects import Reverb, but I find that unlikely.

Some of the effects that I'd potentially like to see:

• Reverb
• Delay
• Chorus/Phaser/Flanger - These mainly have timing differences between them. Correct me if I'm wrong, but they are all essentially delay-based effects with modulation on the playback speed.
• Overdrive/Distortion/Fuzz
• Amp Modeling?
• Pitch Shift (or simply octave up/down)
• EQ/Filter (wah effects could be achieved with a variable BPF)

Something synthio.BlockInput support would be great to have on effect parameters for modulation.

I remember seeing this repo a while back that was able to achieve some decent results with the rp2040: https://github.com/StoneRose35/cortexguitarfx.

[tannewt]

I like the direction of this. I want to point out we probably want more specific module names instead of audioeffects. That way we can have more or less on different chips.