pybricks.tools: Refactor awaitable type. #385

laurensvalk · 2025-09-11T12:36:06Z

This is a rewrite of pb_type_awaitable. This type is responsible for making functions and methods across the Pybricks API async-compatible for use with await.

Adds support for awaiting pbio/os protothreads
This will make it much simpler to await certain interactions with drivers. We'll use this for upcoming changes to the Bluetooth drivers in pybricks/support#2252

Simplified cancellation and reuse
The parent object (e.g. a Motor instanced) used to have a list object to keep track of associated awaitables.

Depending on flags set when a new awaitable is made, it would cancel those before spawning another one. In practice, we only have the "most recent" awaitable that could still be in use, so now we only keep track of that one. We'll re-use it if it has already been fully exhausted and allocate another one otherwise. This also allows unused awaitables to be garbage collected rather than raise a MemoryError if you made too many before awaiting them.

Other awaitable option flags are also removed, because we ended up not using them. Instead, wherever we used PB_TYPE_AWAITABLE_OPT_RAISE_ON_BUSY, it makes more sense to raise the error explicitly in the specific method. For example, I2CDevice.write will raise right away rather than deferring this to the awaitable.

Simplifications for some existing awaitable methods
Awaitable functions that involved more than one operation were very akward to write before, to the point that we ended up writing them in Python and calling them from C code. We don't need to do that anymore. Besides simplicity, this also keep things like cancellation simple.

For example, the awaitable function for hub.speaker.play_notes is now as follows, which is pretty neat. Raising from these functions using MicroPython APIs is allowed, but the return type is still pbio_error_t for compatibility with internal functions.

static pbio_error_t pb_type_Speaker_play_notes_iterate_once(pbio_os_state_t *state, mp_obj_t parent_obj) {
    pb_type_Speaker_obj_t *self = MP_OBJ_TO_PTR(parent_obj);
    mp_obj_t item;

    PBIO_OS_ASYNC_BEGIN(state);

    while ((item = mp_iternext(self->notes_generator)) != MP_OBJ_STOP_ITERATION) {

        // Parse next note.
        uint32_t frequency;
        uint32_t beep_time;
        pb_type_Speaker_get_note(item, self->note_duration, &frequency, &self->scaled_duration, &beep_time);

        // On portion of the note.
        pbdrv_beep_start(frequency, self->sample_attenuator);
        pbio_os_timer_set(&self->timer, beep_time);
        PBIO_OS_AWAIT_UNTIL(state, pbio_os_timer_is_expired(&self->timer));

        // Off portion of the note.
        pbdrv_sound_stop();
        self->timer.duration = self->scaled_duration;
        PBIO_OS_AWAIT_UNTIL(state, pbio_os_timer_is_expired(&self->timer));
    }

    PBIO_OS_ASYNC_END(PBIO_SUCCESS);
}

Raising from these functions using MicroPython APIs is allowed, but the return type is still pbio_error_t for compatibility with internal functions.

Regressions
Bluetooth hasn't been revisited yet, so cancellation of the scan and connect function of remotes is not currently implemented yet.

coveralls · 2025-09-11T12:42:04Z

coverage: 59.255% (+0.7%) from 58.535%
when pulling f14f5ab on async
into 3e8cb1e on master.

github-actions · 2025-09-11T13:06:44Z

Download the artifacts for this pull request:

dlech

Good stuff.

I would feel even better about it if we could add a bunch of unit tests (in Python) that prove this works correctly in all sorts of use cases.

pybricks/common/pb_type_lightmatrix.c

dlech · 2025-09-13T15:14:37Z

pybricks/tools/pb_type_async.c

+/**
+ * Cancels the iterable so it will stop awaiting.
+ *
+ * This will not call close(). Safe to call even if iter is NULL or if it is


Since this doesn't call close(), it sounds more like an abort than a cancel. In computer science, abort means stop it without any additional code executing while cancel means ask it nicely to stop what it is doing and be sure to clean up after yourself and leave things in a consistent state.

For proper cancellation, it is important the async functions can make sure they are not in the middle of an operation that can't be canceled. So it would be problematic to use this with those async functions, e.g. some of the Bluetooth stuff.

I think we should at least document these caveats here and say that this is really only meant to be used in a highly managed way where something else is scheduled to immediately take the place of the aborted iterator. In other words, this only works in the context of pb_type_Task that ensures that close() isn't called on an aborted task.

In other words, this only works in the context of pb_type_Task that ensures that close() isn't called on an aborted task.

These are two different things.

async def main1(): await motor.run_angle(500, very_large_angle) print("we will never see this because the task got closed") async def main2(): await wait(1000) run_task(multitask(main1(), main2(), race=True))

Wait wins the race so the task wraps up and close ensures the motor stops.

async def main1(): await motor.run_angle(500, very_large_angle) print("task not interrupted, just the motor is done here. we do see this") async def main2(): await wait(1000) await motor.run_angle(-500, very_large_angle) run_task(multitask(main1(), main2()))

Here, the motor starts off moving for a long time. But after one second, the same motor is started in reverse from elsewhere. This is safe and fine (new motor command always wins), but we do want the original awaitable to know to stop waiting, which is what pb_type_async_schedule_cancel (or abort) does.

So it schedules the awaitable to cancel the next time it is iterated, so it isn't quite aborted right away. I don't think there is a way to do that, since the caller main1 needs to be the one exhausting it.

I'll include variants of these in the unit tests.

Since this doesn't call close(), it sounds more like an abort than a cancel. In computer science, abort means stop it without any additional code executing while cancel means ask it nicely to stop what it is doing and be sure to clean up after yourself and leave things in a consistent state.

By this definition it is somewhere in between, but I don't mind calling it pb_type_async_schedule_abort.

I guess the right term is pb_type_async_schedule_stop_iteration since that is what it does.

pybricks/tools/pb_type_async.c

pybricks/tools/pb_type_async.h

Can be used to await protothreads, ultimately simplifying most of our awaitable pbio operations.

It turns out that MP_OBJ_STOP_ITERATION is the same as MP_OBJ_NULL, so it can't be used as an intermediate stage. We'll use MP_OBJ_SENTINEL instead and make yield-once functions work another way.

No more linked lists to keep track of awaitables. This fixes programs running out of memory if the user forgets await. Still allow avoiding allocation in basic scripts using a fixed number of statically allocated waits.

This new awaitable is introduced so we can eventually drop queued tasks and safely await protothreads. For now, just keep the pbio tasks compiling.

Now that we can have protothreads as the iterable function, we can trivially do these in C instead of importing Python code from C. Now they can correctly be cancelled, too. A new text animation will take over and stop any ongoing ones.

Also generalize return map to pass in the sensor object. This allows the return map to make use of the sensor state.

Writing both the def and async def for each user method is cumbersome. This allows the user to make a read call and specify how to map the bytes to a return value. This method may then be called as async or sync without further wrappers.

Also flush UART when opening, in case program stopped during prior transmission.

All modules have been upgraded, so we can drop this.

This would shedule cancellation even when already completed. This flagged the awaitable not ready for recycling, so would cause allocation when this isn't needed.

laurensvalk · 2025-09-17T17:38:50Z

Good stuff.

All comments incorporated. Very helpful, thanks!

I would feel even better about it if we could add a bunch of unit tests (in Python) that prove this works correctly in all sorts of use cases.

Good idea. Incoming.

Almost everywhere we use pb_type_async_wait_or_await we stop the ongoing iteration first, so include that functionality as a bool. It is still also available separately for methods that want to stop ongoing awaitables without spawning another one. Also rename pb_type_async_schedule_cancel to pb_type_async_schedule_stop_iteration since cancel isn't quite the right word and this makes it more explicit what it does.

laurensvalk · 2025-09-17T18:42:56Z

I wonder if we can test allocation on the virtual hub? It allocates all the time since it isn't running the compiled mpy.

dlech · 2025-09-17T19:19:10Z

What sort of allocation and testing do you mean? I'm not understanding the question.

laurensvalk · 2025-09-18T06:56:20Z

Having test cases to confirm when it does and does not allocate could be useful.

async def example():
    while True:
        await motor.run_angle(SPEED, 90)
        await wait(500)
        await motor.run_angle(SPEED, -90)
        await wait(500)

For simple scripts like these, this new type ensures that this will not allocate (other than in the very first time it gets to the motor command). This is easy to confirm with mem_info on the embedded ports, but the Virtual Hub is always allocating as it executes byte code.

dlech · 2025-09-18T13:25:15Z

but the Virtual Hub is always allocating as it executes byte code.

That's weird. I would not expect it to. It uses the same runtime after all.

laurensvalk force-pushed the async branch from 68e21e0 to 01ec3b6 Compare September 11, 2025 12:52

dlech reviewed Sep 13, 2025

View reviewed changes

laurensvalk added 15 commits September 17, 2025 18:43

pybricks.tools: Introduce simplified async type.

c4e53ef

Can be used to await protothreads, ultimately simplifying most of our awaitable pbio operations.

pybricks.common.Motor: Use new async type.

2145c9b

pybricks.tools: Allow exhausted awaitable to be re-used.

f96d593

pybricks.tools: Fix awaitable cancellation.

bb9b743

It turns out that MP_OBJ_STOP_ITERATION is the same as MP_OBJ_NULL, so it can't be used as an intermediate stage. We'll use MP_OBJ_SENTINEL instead and make yield-once functions work another way.

pybricks.tools: Use new awaitable for wait.

599e099

No more linked lists to keep track of awaitables. This fixes programs running out of memory if the user forgets await. Still allow avoiding allocation in basic scripts using a fixed number of statically allocated waits.

pybricks.pupdevices: Use new awaitable.

54a72fa

pybricks.robotics.DriveBase: Use new awaitable.

b7d7703

pybricks.tools: Use new awaitable for pbio tasks.

11064fe

This new awaitable is introduced so we can eventually drop queued tasks and safely await protothreads. For now, just keep the pbio tasks compiling.

pybricks.common.Speaker: Use new awaitable.

3d4c5a3

pybricks.iodevices.I2CDevices: Use new awaitable.

9deac01

Also generalize return map to pass in the sensor object. This allows the return map to make use of the sensor state.

pybricks.iodevices.UARTDevice: Use new awaitable.

fbe9752

Also flush UART when opening, in case program stopped during prior transmission.

pybricks.tools: Drop legacy awaitable code.

f676f62

All modules have been upgraded, so we can drop this.

pybricks.tools: Fix cancellation when not active.

9522809

This would shedule cancellation even when already completed. This flagged the awaitable not ready for recycling, so would cause allocation when this isn't needed.

laurensvalk force-pushed the async branch from 01ec3b6 to 0e5b020 Compare September 17, 2025 17:39

tests/virtualhub/multitasking: Add tests for new async type.

f14f5ab

laurensvalk merged commit f14f5ab into master Sep 18, 2025
32 checks passed

dlech deleted the async branch September 18, 2025 13:24

Uh oh!

pybricks.tools: Refactor awaitable type. #385

pybricks.tools: Refactor awaitable type. #385

Uh oh!

Conversation

laurensvalk commented Sep 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

coveralls commented Sep 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Sep 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dlech left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

dlech Sep 13, 2025

Choose a reason for hiding this comment

Uh oh!

laurensvalk Sep 17, 2025

Choose a reason for hiding this comment

Uh oh!

laurensvalk Sep 17, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

laurensvalk commented Sep 17, 2025

Uh oh!

laurensvalk commented Sep 17, 2025

Uh oh!

dlech commented Sep 17, 2025

Uh oh!

laurensvalk commented Sep 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

dlech commented Sep 18, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

laurensvalk commented Sep 11, 2025 •

edited

Loading

coveralls commented Sep 11, 2025 •

edited

Loading

github-actions bot commented Sep 11, 2025 •

edited

Loading

laurensvalk commented Sep 18, 2025 •

edited

Loading