Synthio envelope

[jepler]

This adds a parametric A/D/S/R envelope to synthio. The same envelope is applied to all notes, but can be changed dynamically. the final approach is parametric, though some initial comments describe the original buffer-based approach

envelope = synthio.Envelope(
    attack_time=0.1, decay_time=0.05, release_time=0.2, attack_level=1, sustain_level=0.8
)
melody = synthio.MidiTrack(
    b"\0\x90H\0*\x80H\0\6\x90J\0*\x80J\0\6\x90L\0*\x80L\0\6\x90J\0"
    + b"*\x80J\0\6\x90H\0*\x80H\0\6\x90J\0*\x80J\0\6\x90L\0T\x80L\0"
    + b"\x0c\x90H\0T\x80H\0\x0c\x90H\0T\x80H\0",
    tempo=240,
    sample_rate=48000,
    waveform=sine,
    envelope=envelope,
)

[jepler]

adsr = np.concatenate(
    (
        np.linspace(3277, 32767, num=40, dtype=np.int16, endpoint=False),
        np.linspace(32767, 0, num=20, dtype=np.int16, endpoint=True),
    )
)
#...
synth = synthio.Synthesizer(sample_rate=48000, waveform=waveform, envelope=adsr, envelope_hold_index=42)

A basic linear volume ramp, then slight decay, followed by a sustain at about 80% of full scale. At 48kHz, it takes about 0.2s to ramp to full volume.

The envelope can in principle be changed dynamically, same, as waveform, but I didn't test this.

now tested on pygamer with a variant of my nunchuk instrument program from #7825
I will rebase & fix doc build soon.

[jepler]

looks like the ci docs failure is due to weblate being down for maintenance.

[jepler]

@jedgarpark @todbot in case this is of interest, I'm working on adding ADSR-style envelope to synthio. I think it's ready for somebody to try. However, I'm on vacation 😀 so I won't necessarily act on feedback in a timely fashion. The changes also give another bump in loudness, because I improved (I hope) how the overall envelope of all playing notes is bounded.

[jedgarpark]

@jepler that's great, I'd love to try it out.

[jedgarpark]

@jepler it works, thanks so much! Going to see if i can figure out how to adjust the ADSR values.

[todbot]

This frickin' rules. Thank you @jepler!

[todbot]

adsr = np.concatenate(
    (
        np.linspace(3277, 32767, num=40, dtype=np.int16, endpoint=False),
        np.linspace(32767, 0, num=20, dtype=np.int16, endpoint=True),
    )

Why is the first value "3277"? I would expect it to be zero.  

[jedgarpark]

@jepler this seems to be an ASR envelope, and is working very nicely (or maybe I haven't found the decay time control?) Here's my guess at what the adjustable values are:

attack_time =  40 
sustain_level =  32767
release_time =  140
asr = np.concatenate(
    (
        np.linspace(3277, sustain_level, num=attack_time, dtype=np.int16, endpoint=False),
        np.linspace(sustain_level, 0, num=release_time, dtype=np.int16, endpoint=True),
    )
)
synth = synthio.Synthesizer(
                            sample_rate=48000,
                            waveform=waveform,
                            envelope=asr,
                            envelope_hold_index=42
)

[todbot]

I think this should be a fully configurable ADSR (instead of jepler's original where the decay rate is the same as the release rate), but perceptually the sound seems to "bounce" down-then-up, after the decay. Weird.

attack_time =  100
decay_time = 50
release_time =  300
attack_level = 32767
sustain_level =  5767

adsr = np.concatenate( (
    np.linspace(2, attack_level, num=attack_time, dtype=np.int16, endpoint=False),
    np.linspace(attack_level, sustain_level, num=decay_time, dtype=np.int16, endpoint=False),
    # sustain_level should be at envelope_hold_index = attack_time + decay_time
    np.linspace(sustain_level, 0, num=release_time, dtype=np.int16, endpoint=True),
) )

print("adsr:", adsr[attack_time+decay_time], list(adsr) )

synth = synthio.Synthesizer(
                            sample_rate=48000,
                            waveform=waveform,
                            envelope=adsr,
                            envelope_hold_index=attack_time+decay_time
)

This is really cool! My questions thus far:

• Is this "bounce" just my ears or am I doing something wrong?
• What are the units of this "attack_time", "release_time", etc?
• Why the 3277 for the first entry in the list? Looks like I can use 300 or 30 or 2, but 0

[jepler]

• If the first element of ADSR is 0, it's just like starting the sound 1 time-unit later, so start with nonzero. 3277 was left over from when the attack portion was 10 units long, so it's about 32768/10.
• it is ADSR, not ASR, because you can set the "envelope hold index" to set a sample within the envelope array which is used during the sustain portion. So for instance if you have the very simple ramp [10000, 20000, 30000, 20000, 10000] and set the hold index to 2 (indicating the element with value 3000) you get ASR. If you set the hold index to 3 (indicating the element 20000) you get ADSR, with a relative volume of 20000 during the release phase
• The time units are groups of 256 samples, so at 48kHz one element is about 5ms. At 24kHz an element is about 10ms, etc.
  • except in MidiFile, where any note-change ALSO causes a step through the ADSR waveform, which is a bug/limitation
• I'm not sure what's going on with a perceived "bounce". The overall handling to allow one voice to be loud but preventing multiple voices from clipping is probably creating this effect. Does anyone have technical resources describing how to do this properly from a coding POV?

vertical line indicates hold (sustain) index:

which could end up like this if the note is sustained for a time:

clearly the option should be renamed from "hold index" to "sustain index".

[jepler]

# Time is in units of 256 samples, e.g., ~5ms at 48kHz                          
attack_time = 40                                                                
peak_level = 30000                                                              
decay_time = 40                                                                 
sustain_level = 20000                                                           
release_time = 140                                                              
                                                                                
adsr = np.concatenate(                                                          
    (                                                                           
        np.linspace(                                                            
            peak_level / attack_time,                                           
            peak_level,                                                         
            num=attack_time,                                                    
            dtype=np.int16,                                                     
            endpoint=False,                                                     
        ),                                                                      
        np.linspace(                                                            
            peak_level, sustain_level, num=decay_time, dtype=np.int16, endpoint=False                                                                           
        ),                                                                      
        np.linspace(sustain_level, 0, num=release_time, dtype=np.int16, endpoint=True),                                                                         
    )                                                                           
)                                                                               
sustain_index = attack_time+decay_time

[jepler]

renamed the parameter to envelope_sustain_index

[todbot]

Thanks @jepler, that makes sense. This is a lot of fun to play with and already so very useful.

Some comments from trying it out:

• On key release, the envelope currently plays from envelope_sustain_index which causes a jump in loudness if the note is still ramping up during attack. (I think that's what it's doing, maybe it's my velocity code?) The more expected behavior would be to play from equivalent loudness.

• Synths often use MIDI velocity to change the amplitude envelope: both peak/sustain levels and attack & release rates. e.g. play softly to get a quiet slow rise and fall of the sound, play hard to get a fast, loud sound. I am unsure the best way to accomplish this. What I have been doing is declaring a max number of time units to be divvied up among the attack, sustain, and release parts of the envelope. Calculating this new envelope is kind of a hassle but I think possible. Does this sound like the appropriate way to do this? You can see one implementation I've done here: simple_synthio_midi_w_velocity.py

• There is a .press() method but no .release() method. This means when making a standard MIDI-style synth one has to do synth.release_then_press( release=(msg.note,), press=() ) to do a MIDI noteOff. Not a big deal, but having matching press/release methods would match MIDI more.

I've been putting my experiments up as gists. Thank you for putting yours up their too. I loved the wind chime one!

[jepler]

Yes, right now releasing just jumps from whatever spot in the envelope to the first index after the sustain index. Given the underlying table-based approach I chose, what else would be sensible to do? Or should I abandon the table-based approach? (it's not too late for that)

I'd like to directly support velocity (as relative note volume), but I don't know what the API would look like. Notes would either become objects with multiple properties, or all the 'parts' would be packed together into a single number like e.g., 24-bit rgb values are. I'd like to pick something that would work when doing multiple 'channels' (independent waveforms & envelopes). Are the NoteOn and NoteOff classes from adafruit_midi something to follow here? Or should I literally move closer to parsing a midi steam, like with a literal write method that takes midi bytes?

[todbot]

Given the underlying table-based approach I chose, what else would be sensible to do? Or should I abandon the table-based approach? (it's not too late for that)

Oh, if the API is still fluid, I have opinions! 😀

The current waveform approach is interesting and allows for complex envelopes, so it'd be great for LFOs, but I don't see how to make it work easily as an ADSR envelope, where the envelope can constantly change based on user playing.

Most of the synth APIs I have come across have something akin to "setLevels(v1,v2,v3,v4)" and "setRates(r1,r2,r3,r4)" methods on an envelope. The envelope system lerps between the levels according to the rates. So the envelope system tracks only these 8 values and the one or two internal values for lerping.

Generally the above is wrapped up in an "Envelope" object and it's the Envelope that contains a "noteOn()" method to start the envelope attack and a "noteOff()" method to switch to the release phase. (There is a separate "Oscillator" object that is always running and contains a "setFrequency(hz)" method. This is so one can devote multiple detuned oscillators per played note, a common trick to make more complex sounds, something I think we can emulate with wavetables)

Pseudo-code using such a design handling MIDI messages might look like:

if( msg == noteOn ) {
  oscillator.setFrequency( midi_to_hz(msg.note) )
  envelope.setRates( attack=msg.velocity*2 ) // fast attack on hard noteOn
  envelope.setLevels( sustain=msg.melocity*2 ) // max sustain loudness on max velocity
  envelope.noteOn()
}
else if( msg == noteOff ) {
  envelope.setRates( release=255-msg.velocity*2 ) // slow release on light noteOff
  envelope.noteOff()
}

For a fairly easy-to-read example of this kind of API, see Mozzi's ADSR.h and Oscil.h.

I can imagine a synthio shape of the above taking envelope_levels and envelope_rates ReadableBuffers params. The user would adjust envelope_levels[2] to change the sustain level and envelope_rates[0] to change attack rate.

I'd like to directly support velocity (as relative note volume), but I don't know what the API would look like. Notes would either become objects with multiple properties, or all the 'parts' would be packed together into a single number like e.g., 24-bit rgb values are. I'd like to pick something that would work when doing multiple 'channels' (independent waveforms & envelopes).

I think this kinda sounds like the two buffer params I was imagining too. I wish there was a way to modularize this a bit more so that Synthesizer wasn't required to hold on to so much state by itself. But I don't understand how it could be modularized either.

Are the NoteOn and NoteOff classes from adafruit_midi something to follow here? Or should I literally move closer to parsing a midi steam, like with a literal write method that takes midi bytes?

I would follow neither, honestly. The adafruit_midi class is super wordy for the simplicity of the 3-byte protocol of MIDI. And besides, MIDI is a rather flat representation of the richness of audio synthesis. It's like sheet music. It's a good source of "triggers" for an already-configured synthesizer, but how that synth is configured (its oscillators, envelopes, LFOs, filters, etc) has never been the domain of MIDI.

Also standard MIDI assumes a Western 12-tone equal temperament scale, great for most uses, but some people might want to do just temperament or microtonal scales. So if possible, I would recommend the ability to set frequency directly in Hz.

[tannewt]

I can imagine a synthio shape of the above taking envelope_levels and envelope_rates ReadableBuffers params. The user would adjust envelope_levels[2] to change the sustain level and envelope_rates[0] to change attack rate.

I'd prefer 8 separate parameters with better names. indices are hard to understand.

[todbot]

I'd prefer 8 separate parameters with better names. indices are hard to understand.

Me too. But for some reason I was assuming we need to stick to ReadableBuffers to allow real-time control of these params. I am probably wrong about that.

[dhalbert]

Maybe have a separate data class that holds config params, or use a namedtuple?

[jepler]

I think there's no reason that synth.envelope = synthio.Envelope(attack_time=37, ...) couldn't be made to work.

If it's what y'all want I can abandon the arbitrary-buffer-based way of this PR as it stands, and switch to something more like the mozzi's envelope generator.

switching synthesis to be based on frequency in (possibly floating point) Hz instead of MIDI notes would be a topic for a future PR.

[jepler]

@todbot in the mozzy way what parameters define a 'plucked' note's envelope?

[jepler]

Here's what I tentatively have (only this draft documentation) for a more mozzy-like envelope (and the Synthesizer object will have a runtime-assignable envelope property):

class synthio.Envelope(attack_time: float, decay_time: float, release_time: float, attack_level: float, sustain_level: float)`

Construct an Envelope object

The Envelope defines an ADSR (Attack, Decay, Sustain, Release) envelope with linear amplitude ramping. A note starts at 0 volume, then increases to attack_level over attack_time seconds; then it decays to sustain_level over decay_time seconds. Finally, when the note is released, it decreases to 0 volume over release_time.

If the sustain_level of an envelope is 0, then the decay and sustain phases of the note are always omitted. The note is considered to be released as soon as the envelope reaches the end of the attack phase. The decay_time is ignored. This is similar to how a plucked or struck instrument behaves.

If a note is released before it reaches its sustain phase, it decays with the same slope indicated by sustain_level/release_time (or attack_level/release_time for plucked envelopes)

Parameters:

• attack_time (float) – The time in seconds it takes to ramp from 0 volume to attack_volume

• decay_time (float) – The time in seconds it takes to ramp from attack_volume to sustain_volume

• release_time (float) – The time in seconds it takes to ramp from sustain_volume to release_volume. When a note is released before it has reached the sustain phase, the release is done with the same slope indicated by release_time and sustain_level

• attack_level (float) – The relative level, in the range 0.0 to 1.0 of the peak volume of the attack phase

• sustain_level (float) – The relative level, in the range 0.0 to 1.0 of the volume of the sustain phase

attack_time_: float_¶

The time in seconds it takes to ramp from 0 volume to attack_volume

decay_time_: float_¶

The time in seconds it takes to ramp from attack_volume to sustain_volume

release_time_: float_¶

The time in seconds it takes to ramp from sustain_volume to release_volume. When a note is released before it has reached the sustain phase, the release is done with the same slope indicated by release_time and sustain_level

attack_level_: float_¶

The relative level, in the range 0.0 to 1.0 of the peak volume of the attack phase

sustain_level_: float_¶

The relative level, in the range 0.0 to 1.0 of the volume of the sustain phase

does this make sense / sound like something that is better than the current table-based approach?

[tannewt]

If it's what y'all want I can abandon the arbitrary-buffer-based way of this PR as it stands, and switch to something more like the mozzi's envelope generator.

I'm ok with a buffer if it is a shape. I'd avoid a buffer if you'd want each index to mean something else. In that case, it should be named values.

(I leave the synth decisions to folks who've done it.)

[todbot]

@todbot in the mozzy way what parameters define a 'plucked' note's envelope?

This is the problem with ADSR envelopes in general. Many synths solve the problem by defining a curve shape (linear, exp, inv exp) to the rates. But the synth libraries I know only do linear rates for their ADSR (e.g. Mozzi, TeensyAL, DaisySP are linear).

The DaisySP library has an separate AD envelope w/ a curve setting that is meant for percussive sounds like drums and plucks. Mozzi has something similar with an "EAD" envelope.

If it's easy to bake in a slight exponential-like decay, I think that would be the most musically useful without needing to provide yet-another API parameter. Most sounds decay like that so it's what humans expect.

[todbot]

Here's what I tentatively have (only this draft documentation) for a more mozzy-like envelope (and the Synthesizer object will have a runtime-assignable envelope property):

class synthio.Envelope(attack_time: float, decay_time: float, release_time: float, attack_level: float, sustain_level: float)`
[...]
does this make sense / sound like something that is better than the current table-based approach?

Makes a lot of sense to me. And maybe opens the door for potentially other kinds of envelopes that people will want.

[jepler]

OK, I'll re-work this PR so that it uses a parameter-based envelope instead of a table-based one. Thank you for the input.

[todbot]

Thanks @jepler! I am astounded that we can get polyphonic synth functionality in CircuitPython.

[jepler]

This is updated! Now an envelope can be used similar to this, from my testing code:

envelope = synthio.Envelope(
    attack_time=0.1, decay_time=0.05, release_time=0.2, attack_level=1, sustain_level=0.8
)
melody = synthio.MidiTrack(
    b"\0\x90H\0*\x80H\0\6\x90J\0*\x80J\0\6\x90L\0*\x80L\0\6\x90J\0"
    + b"*\x80J\0\6\x90H\0*\x80H\0\6\x90J\0*\x80J\0\6\x90L\0T\x80L\0"
    + b"\x0c\x90H\0T\x80H\0\x0c\x90H\0T\x80H\0",
    tempo=240,
    sample_rate=48000,
    waveform=sine,
    envelope=envelope,
)

[jepler]

@jedgarpark @todbot I think this is ready for another round of testing.

[todbot]

Hi @jepler!
I finally got around to trying this out. Very cool! Much easier to change envelope rates based on incoming note playing, and we can now make some pretty expressive synths.

A few observations:

• The synthio.Envelope is "global" to the synth, so the amplitude envelope of currently playing notes gets replaced by notes with a different envelope (e.g. play gentle, then play hard, hard notes' env replace gentle notes' env). It would be nice to have per-note envelope state, but that may be asking too much. :)
• It seems natural to say amp_env.attack_rate = 1.2 (I mean amp_env.attack_time) but that's a read-only property. Not a big deal, just noticed it.
• Playing two simultaneous notes seems obviously quieter than playing one, and quieter again playing three

Here's a gist with a demo video showing what I've been testing with.
https://gist.github.com/todbot/9bbbcd93e04c9ce258b0f4ffbe7dc43a

[jepler]

• The synthio.Envelope is "global" to the synth, so the amplitude envelope of currently playing notes gets replaced by notes with a different envelope (e.g. play gentle, then play hard, hard notes' env replace gentle notes' env). It would be nice to have per-note envelope state, but that may be asking too much. :)

I want to tackle that in the next round of improvements, so that a note can be associate with an envelope rather than having just one envelope for the whole Synthesizer object.

• It seems natural to say amp_env.attack_rate = 1.2 but that's a read-only property. Not a big deal, just noticed it.

The current implementation of Envelope objects makes all properties read-only. I think this may have to stay as it is, because otherwise it gets complicated to decide what notes would be affected (need something re-calculated internally) when assigning a property. I'm open to ways around this so I can lift the restriction in the future.

• Playing two simultaneous notes seems obviously quieter than playing one, and quieter again playing three

Yes. This is true.

I remain open to finding some different way to avoid clipping (or, I guess, permission to say that it's the user's fault when they get clipping).

The number of simultaneous voices is fixed at compile time, but a particular user may need fewer. Maybe just one. Another user may want all 12, or even wish for more. Furthemore, the 1-voice user will want that voice to be able to play at full loudness, not 1/12 loudness. So, the synthesizer implements a very aggressive form of dynamic range compression by taking the sum of all the current notes' envelope loudness, and dividing the overall loudness by min(1, sum_envelope_loudness). This value is re-calculated every 256 samples, the same interval with which envelope calculates are updated.

[tannewt]

A few minor things. Very cool that you have tests for this and can run it from the Unix port.

[jedgarpark]

Testing envelope now -- this is working really nicely, sounds great! I think the envelope parameters make a lot of sense now (to people familiar with typical synth methods at least).

[jepler]

@microdev1 this is failing in a weird way:

Run python3 -u ci_changes_per_commit.py
Traceback (most recent call last):
  File "/home/runner/work/circuitpython/circuitpython/tools/ci_changes_per_commit.py", line 239, in <module>
    main()
  File "/home/runner/work/circuitpython/circuitpython/tools/ci_changes_per_commit.py", line 228, in main
    check_runs = get_bad_check_runs(query_check_runs)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/circuitpython/circuitpython/tools/ci_changes_per_commit.py", line 191, in get_bad_check_runs
    matrix_job = name.rsplit(" (", 1)[1][:-1]
                 ~~~~~~~~~~~~~~~~~~~~^^^
IndexError: list index out of range
Error: Process completed with exit code 1.

please let me know if there's something I can do to resolve it

[jepler]

@microdev1 the CI problem seems to have healed so that's good I guess

[tannewt]

One nitpick that you can do in your next PR if you like.

[tannewt]

This no longer prints. Maybe set, record or report?