synthio: add sequencer functionality

[jepler]

For example, given a suitable definition of laser_note and laser_lfo, this sequence will play a laser sound for around 3/10 of a second:

laser = [
   {"press":laser_note, "retrigger":laser_lfo},
   0.3,
   {"release":laser_note}
]

this may not even work (I simply dashed it off while at the airport) but it shows the general idea of what @deshipu and I talked about at PyConUS. ping @jedgarpark @todbot

The same foundation would seem to enable basic drum machine style sequencing where a pattern is repeated over time ... except that I didn't actually implement looping, oops. Instead, there's a hack (that I didn't even test) that should allow a sequence to self-retrigger:

looping_laser = [
   {"press":laser_note, "retrigger":laser_lfo},
   0.3,
   {"release":laser_note}
]
looping_laser.append({"start": looping_laser})

since a sequence is just calls to press, the loop can re-trigger itself if its last element requests it. I do not want to ship this as the way to loop a sequence though.

This also removes the deprecated release_then_press method, which is a subset of change functionality.

[todbot]

Neat! What is the retrigger key for and why does it take an LFO?

[deshipu]

Neat! What is the retrigger key for and why does it take an LFO?

It resets the LFO to the initial state.

[todbot]

Neat! What is the retrigger key for and why does it take an LFO?

It resets the LFO to the initial state.

I see. This mirrors the Synthesizer.change() API (which I don't use much)

[todbot]

Would this new functionality be appropriate for automating filter changes too? Filter modulation currently has to be done by hand, creating a detached LFO on note press, reading it repeatedly and applying it to Synthesizer.low_pass_filter(). I suspect not, but thought I'd ask. 😀

[deshipu]

This is completely wild fantasizing, just to see what we could do.

In theory, there is no reason to make all the changes in one step, because as long as there is no delay in the list of steps, all steps should execute at the same time. So maybe we could simplify it by allowing a Note and an LFO to appear as a step, alongside the Float, instead of dicts? The LFO would get re-triggered then, but there is some ambiguity about what should happen with the note - we need some way of marking if it should be started or stopped. I don't suppose we could make the unary minus work on the Note object easily, and return a "stop" version of it?

Then it would look something like:

laser = [laser_note, laser_lfo, 0.3, -laser_note]

If the unary operator is not possible, then we could wrap it explicitly:

laser = [laser_note, laser_lfo, 0.3, StopNote(laser_note)]

[todbot]

laser = [laser_note, laser_lfo, 0.3, -laser_note]

This seems more confusing to me, if the intent of the sequence data struct is meant to represent a timed set of calls to Synthesizer.change()

[deshipu]

And I think that for sanity's sake we should also automatically stop all notes at the end of the sequence.

[deshipu]

I also have slight problem with putting the start and stop commands in the synth.change function. Arguably, they are "higher level" operations than the press, release and retrigger, and I already confused start with press and stop with release. I know that it's convenient from the C code point of view to have everything done with one function with a lot of different keyword arguments, but it could get out of control pretty fast.

[deshipu]

Do we even still need the change method when we can trigger events simultaneously with sequences now?

[todbot]

To me, this feature currently seems identical to an async function we can currently write, like:

async def play_laser_sequence():
  synth.change(press=laser_note, retrigger=laser_lfo)
  await asyncio.sleep(0.3)
  synth.change(release=laser_note)

I really like the idea of a "list of change() commands` as a basis for a sequencer. It's simple to understand. But a common use of a sequencer is to set it playing in a loop and modifying the contents as is plays. So I feel it should have at least:

• looping can be set
• tempo can be adjusted on the fly
• note on-time can be adjusted on the fly

If we can adjust the sequence data struct without resetting it, like we can with Synthesizer.waveform, I think that would enable the above functionality.

[deshipu]

It's easy to understand for you, because you already know how the change function works, and feel perfectly at home with the idea that keyword arguments are the same thing as a dict. It's harder to explain this to new users, especially if the change method is removed anyways (because the same thing can now be done with sequences).

Other users might be more familiar with the interface already provided by the audio*io.AudioOut, audiomixer.Mixer and audiomixer.MixerVoice where you have a play(..., loop=False) method, together with playing, stop, pause, resume and paused methods and properties. Yes, it will take up more space in the flash, but it would be much more consistent with existing libraries. If space is a concern, we could skip the pausing.

[todbot]

It's easy to understand for you, because you already know how the change function works, [...]

Well to be fair, until today I had totally forgotten .change() existed because I only use .press() and .release(). And if you were to ask me yesterday "What should a sequence data structure look like for synthio?" I would've said something like "List of NoteOn and NoteOff messages, just like MIDI". I feel anyone using a sequencer concept has to be conversant with this base level understanding to get anything done. Jepler's initial proposal is very similar.

I don't really care what the data structure looks like as long as it is an event list that can edited while the sequence is looping.

If you're advocating for a Synthesizer.play(sequence) method, instead of overloading .change(), that would make a lot of sense.

[deshipu]

So something like:

laser = [NoteOn(laser_note), laser_lfo, 0.3, NoteOff(laser_note)]
synth.play(laser, loop=True)

for example?

[jepler]

Happy to use a different approach, though new types seem to chew up a bunch of flash.

Yes this is not a solution to changing filters dynamically.

[jepler]

Compared to the async routine this code guarantees changes happen atomically when respect to sample generation and that delays are consistent. Also it never generates garbage.

[deshipu]

Happy to use a different approach, though new types seem to chew up a bunch of flash.

Let's consider using play method with the sequence of dicts:

laser = [{'press': laser_note}, {'retrigger': laser_lfo}, 0.3, {'release': laser_note}]
synth.play(laser, loop=True)

I can see several potential inconveniences here:

• The strings are arbitrary. If I were to come up with a verb describing starting a note, I don't think I would come up with "press". I suppose this is the legacy of MIDI and of it being practically a piano-centric protocol, but it might be a problem.
• Using the wrong strings or making typos in them doesn't lead to errors or any kind of syntax highlighting help. It simply doesn't work without any hint about what is wrong. Typos will result in TypeError with a message about function taking wrong arguments, I suppose?
• It's a bit more verbose than it could be.

Having said that, it all doesn't matter for my particular use case, because it will be all generated by the library, and users will never have to look inside. So, while I could imagine a more elegant format, this works for me too.

[todbot]

Agreed @deshipu. The dict originally proposed is more verbose than I'd like. What about a list where "element 0 is 'pressed', element 1 is 'released', element 2 is 'retrigger'" (i.e. matching .change()'s positional args):

laser = [ [laser_note, None, laser_lfo],   # press, release, retrigger
          0.3,
          [None, laser_note, None] ]  # press, release, retrigger

Hmm that looks even more dangerous. Would namedtuple be about the same cost as as dict?

Regardless of the format, what I really would like is some sort of in-place editing of the submitted sequence. E.g. let's say I have a simple bass drum pattern that goes "boom-boom----bom-boom", at a given tempo, and I want to slow it down 10%. Can I do:

drum_bd_seq = [
    {'press': bassdrum_note},
    0.001,  # minimum time to trigger drum
    {'release': bassdrum_note},
    0.5,   # 500 ms = 120 bpm
    
    {'press': bassdrum_note},
    0.001,
    {'release': bassdrum_note},
    0.5,
    
    {'press': bassdrum_note},
    0.001,
    {'release': bassdrum_note},
    0.75,
    
    {'press': bassdrum_note},
    0.001,
    {'release': bassdrum_note},
    0.25,
]

synth.play(drum_bd_seq, loop=True)

# slow down tempo 10% of playing loop by changing all note delays
# (which we've designed to be every 4th element)
for i in range(3, len(drum_bd_seq), 4):
    drum_bd_seq[i] *= 1.1

Note that we have some surprises we have to contend with:

• Sequence length/timing is determined by sum of delays
• Must be sure to pair a 'release' with every 'press'
• To change tempo, must only change between-note delays (and know where they are in the list)
• Delay between press & release needs to be factored into total sequence length when changing tempo (not done in above example)

But on the upside, we can swap out notes to play to seamlessly alter the 'feel' of a sequence and (with work) the speed of the sequence to match the situation (very useful for games)!

The way most sequencers deal with some of these issues is to have a "gate time" parameter that specifies how long a note should be held. That is, instead of a note event consisting of three objects "NoteOn, Delay, NoteOff", it consists of one object "NoteEvent(note_val,gate_time)" and the sequencer keeps track of playing notes and their off times. This is perhaps too much to expect of synthio so I wasn't proposing it, but it does make timing adjustments a lot easier. (For instance, changing all NoteEevent gate_times while playing let's you seamlessly go from playing staccato to legato)

[dhalbert]

I will say a few things here:

You are inventing a (tiny) programming language. I have seen this a lot in various contexts. A question I always ask is whether you need a new language, or do you have one already? We do, and @todbot points out how to use Python and async to do things. If the primitives are not want you want, they could be changed.

Assuming you still want to make a language:

A dict or a named tuple is a kind of verbose way of doing this, and a bit expensive. There's a fixed set of strings above, so you could use an enumeration (i.e., just integers) to represent what things mean. That would be compact. The integers could be constants in a synthio class or synthio itself, or they could be added in a python helper library.

The alternations of event and time seem a little weird to me. The time values above are just another operation: delay. Each thing in the sequence could be a tuple

I would say try writing this more abstractly first (e.g., in Lisp or some other abstract notation), and then decide how to represent it.

[deshipu]

@todbot this looks terribly complex and fragile to me. Again, that's not a concern for me, because in my use case the representation would be hidden from the user, but just for the sake of brainstorming, what would happen if duration became an attribute of the Note?

We wouldn't need a release anymore, because every note would be automatically released when its duration expires. We wouldn't need a delay (we could have an "empty" Note for pauses) anymore either. We could then simply just have a sequence of Notes, and changing their timing would be just a matter of iterating over them and changing their duration, or maybe inserting some additional pauses in between them.

The only problems that appear then are about playing several notes at once, and about re-triggering the LFO. Internally, those problems are solved by using channels. So what if we could specify multiple sequences, one per channel. (And possibly one extra for the LFO? I don't really have enough experience with them to tell if we really need this kind of control over them, or if re-triggering them when the associated Note starts would suffice.) This also seems to be how music notation tends to be written, with a separate staff for each hand on the piano, and how music trackers show you the input, so hopefully this would be familiar enough?

[deshipu]

@dhalbert Are you suggesting that we should eventually (once we know how we want the language to look like) go the way of the PIO assembly, and just specify a text string with the language in it, and parse it into whatever internal structure we need?

[todbot]

I've wanted a better musical timing facility than is currently available at the user level. CircuitPython is unusable in many musical contexts because of its sloppy timing. Even a "timed list of synth.change() calls" as currently proposed is better than what we can do now. And as DSLs go, is pretty simple. I'd very much welcome a richer one though.

But to me editing of the submitted list is key: without that, it's basically the same as the fire-and-forget nature of audio.play(WaveFile("laser.wav")), and if so, I'd rather just play a WAV.

[dhalbert]

@deshipu No, no, I'm saying you're inventing a programming language that's represented as a data structure. Let's see if we need a new language, and if so, what are its primitives, and then see how to represent that language. The language representation does not need to be text.

[deshipu]

@dhalbert It's a data format, for it to be a programming language, I would expect it to have control structures. Specifically, it's a time series data.

[dhalbert]

I'm not saying it's a Turing complete language. I'm saying that conceptually it specifies operations and their arguments. And I could easily imagine it starting to have conditionals, loops, etc. By my definition MIDI sequences are a programming language, yes. Maybe you'd disagree.

[todbot]

We wouldn't need a release anymore, because every note would be automatically released when its duration expires. We wouldn't need a delay (we could have an "empty" Note for pauses) anymore either. We could then simply just have a sequence of Notes, and changing their timing would be just a matter of iterating over them and changing their duration, or maybe inserting some additional pauses in between them.

You got it. This is what most sequencers do: store NoteEvents with a gate_time (e.g. duration) and events fall on time steps in a sequences (usually called ticks). So the above sequence instead would look like (in an imagined world of (time,event) tuples and 24 ticks per quarter note) :

drum_bd_seq = 
[   # tick time, event
    [0,  NoteEvent(BassDrum, duration=0.1)],
    [24, NoteEvent(BassDrum, duration=0.1)],
    [72, NoteEvent(BassDrum, duration=0.05)],
    [84, NoteEvent(BassDrum, duration=0.1)],
]

[deshipu]

In so far as programming is telling the computer what to do, it's programming, for sure. But the thing that we want to express is ultimately a static piece of data – what notes should be played with what effects and what timings. The temporal nature of the data does make it look like a set of operations, but that's not the only representation possible. After all, you can always describe static data by (one possible) procedure leading to producing that data. Like you can describe an image by a sequence of drawing operations. But the final image is usually simpler, and we could probably do better.

[deshipu]

Perhaps instead of putting this in synthio directly, we could only have a minimal, bare-bones way of playing a sequence of notes in the background, designed to be minimal, and have a separate sequencer module written in Python that uses a user-friendly interface?

[jedgarpark]

I don't have opinions on what constitutes a programming language, but i would love to be able to create something like nanoloop in CircuitPython to run on a PyBadge/PyGamer https://youtu.be/Kxdgo86mSW0?si=zvaASdwh1RzFwVB9&t=306

[jepler]

This is a bit of a brain dump, sorry!

For me the reason to use a data structure instead of Python code is because the "evaluation" is occurring in a C background task, at a moment when we need to provide more audio samples under a deadline or playback will glitch. We routinely have trouble when we try to allow the C background task call back out to general Python code. (that said, if you're evil you can supply a python generator with the current implementation; when this breaks you get to be sad)

I chose to make it in the form of "calls to change(**kwargs)" for two reasons: first, because it meant the amount of core code to add was small, and because I figured "synthio people need to learn how change works, and then they can apply that knowledge in a new context". If we end up with a radically different way to do sequences, I would want to turn around and make that the main way to do a "dumb operation".

Explicit note durations: This is interesting. Due to the way synthio grew organically out of MIDI, which didn't have explicit note durations (to my limited knowledge), this isn't supported.

Time stamps: I suppose they COULD be relative to the start of the sequence. Does this make anything better/worse on the whole? Related question: is there a better "unit" than seconds?

Memory size: Is list vs dict important? The size difference is not much. It's only convenient to measure the sys.getsizeof() the objects on the unix coverage build, where all pointers are twice as big as our 32-bit ARMs. But, in case it's a useful proxy measurement, a dict with 2/3 elements is 64/80, while a list with 3 elements is 60 bytes:

>>> sys.getsizeof([3, 3, 3])
64
>>> sys.getsizeof({"press": 3, "release": 3, "retrigger": 3})
80

I'm going to mark this PR as closed, because the implementation I proposed is clearly on the wrong track from the point of view of my fellow commenters. That's not intended to stop the conversation, just to remove it from the list of open PRs.

[deshipu]

Time stamps: I suppose they COULD be relative to the start of the sequence. Does this make anything better/worse on the whole? Related question: is there a better "unit" than seconds?

This one I know! At least for music, you want "beats", with a tempo specified globally, and then one beat being one "tick" in that tempo. Fractional beats would be rare.

[todbot]

In MIDI, the timing resolution is "ticks" or "pulses" with 24 PPQ (24 pulses per quarter note) being the most common. (see MIDI beat clock)
I don't think this is necessarily required for synthio though, seconds is fine.

Overall I thought the "list of calls to synth.change()" was fine and assumed we'd build user-level sequencer classes that wrapped that and converted from a more human-readable format.

[deshipu]

@jedgarpark I think this is already possible!

[todbot]

@jedgarpark I think this is already possible!

It is sorta. I built it. And the timing is too sloppy to be useful. I ended up porting it all to Arduino.

[deshipu]

@todbot Same for me, the only thing I didn't like is adding of the "start" and "stop" operations, because that's different from what other similar modules do.

But if we assume there will be a Python module that will give us a more convenient interface (as I assumed for my sound effect library), then the exact format doesn't really matter.

[jedgarpark]

@deshipu @todbot yes, the timing needs to be rock solid for this, and i'm thinking the use of four synthio voices so it is a sequencer with built in synthesizer that outputs audio would be more taxing than sending out MIDI.

[jepler]

Without a major change to the architecture of synthio, timing ultimately will refer to groups of 256 samples; 5.805…ms at 44.1kHz, for instance. Whether the duration is a fractional second or a (possibly fractional) MIDI beat, it'd be rounded. So say you're at 132bpm -- each beat will be 78.303… sample groups, or, put another way, you could actually achieve 132.51…bpm. (and then you get 78=2x3x13 subdivisions, which can't support equal 16th notes, they'd have to be alternating between 19 and 20 units)

I don't really WANT to expose this detail but it feels like it'll have to be exposed.

So say you're building up your hot new techno track in stages, you need to be able to bring in a new element at will but have it synch'd to the running elements. which almost points at needing to be able to say: "start this sequence at sample block #N". or something.

Feeling like I bit off something too big to chew, here.

[samblenny]

Just saw this discussion mentioned in the weekly meeting notes. FWIW, I've been off in my corner for the past week and a half tinkering with a CircuitPython sequencer idea. I'm building a thing to translate plaintext song notation files into timed MIDI messages over USB MIDI.

For song notation, I'm using a text file format that borrows ideas from the abc music standard for describing note sequences in terms of accidental, pitch, octave, and duration. The full abc standard is very complicated with a grammar that would be challenging to parse, so I'm not trying to actually implement the standard. But, it might be a useful source of ideas on how to concisely describe music as ASCII text.