Add async/await syntax to FULL_BUILD

[kvc0]

This adds the async def and await verbs to valid CircuitPython syntax using the Micropython implementation.

Consider:

>>> class Awaitable:
...     def __iter__(self):
...         for i in range(3):
...             print('awaiting', i)
...             yield
...         return 42
...
>>> async def wait_for_it():
...     a = Awaitable()
...     result = await a
...     return result
...
>>> task = wait_for_it()
>>> next(task)
awaiting 0
>>> next(task)
awaiting 1
>>> next(task)
awaiting 2
>>> next(task)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  StopIteration: 42
>>>

and more excitingly:

>>> async def it_awaits_a_subtask():
...     value = await wait_for_it()
...     print('twice as good', value * 2)
...
>>> task = it_awaits_a_subtask()
>>> next(task)
awaiting 0
>>> next(task)
awaiting 1
>>> next(task)
awaiting 2
>>> next(task)
twice as good 84
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  StopIteration:

Note that this is just syntax plumbing, not an all-encompassing implementation of an asynchronous task scheduler or asynchronous hardware apis.
uasyncio might be a good module to bring in, or something else - but the standard Python syntax does not strictly require deeper hardware
support.
Micropython implements the await verb via the iter function rather than await. It's okay.

The syntax being present will enable users to write clean and expressive multi-step state machines that are written serially and interleaved
according to the rules provided by those users.

Given that this does not include an all-encompassing C scheduler, this is expected to be an advanced functionality until the community settles
on the future of deep hardware support for async/await in CircuitPython. Users will implement yield-based schedulers and tasks wrapping
synchronous hardware APIs with polling to avoid blocking, while their application business logic gets simple await statements.

[kvc0]

While we wait for the list of microcontrollers that are too micro to async, here's an example that shows how exception propagation is able to be handled in an async/await context.

>>> async def it_throws():
...     yield
...     raise Exception('ack')
...
>>> async def it_catches():
...     try:
...         await it_throws()
...     except Exception as e:
...         print('ah ha, I caught', e)
...
>>> task = it_catches()
>>> next(task)
>>> next(task)
ah ha, I caught ack
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
StopIteration:

[kvc0]

And here's an async lib I threw together for this: https://github.com/WarriorOfWire/circuitpython-utilities/tree/master/budget_async

You can use it kind of like an asyncio event loop. This was really easy to put together; there is much opportunity for others to make better things.

It ties async/await to an event loop that you'd most likely call next() on in a tight while: True loop in your main application function. While in a coroutine you can await, you can await loop.sleep() and you can wrap blocking device libraries that at least provide a polling mechanism behind an apparently concurrent api, allowing other code to run while you wait for things like sensor responses and network activity. Thanks async! You're a pal.

[jepler]

Do any of your examples work in both the modified CircuitPython and in Python 3? I was not able to modify your examples to work in both interpreters.

[kvc0]

No; Micropython implemented the async verb as producing a generator. Python 3 uses a coroutine instead. The apis are not the same.

This impacts developers of "nuts and bolts" features, like that budget async lib, but it should not impact regular use. I.e., if you have a suitable concurrency lib in your project, you just use it and can choose to write methods with async/await.

I'm aiming to help us get mileage on the syntax, not specifically to Solve The Problem universally. I'd like to switch up the implementation a little to be closer to CPython's but allowing the syntax and trying it out is the first step.

[tannewt]

@WarriorOfWire Thanks for making a PR for this! It's nice to see a PR proposal.

I don't want to introduce an async API that is incompatible with CPython because then code will be built on top that we'd have to break later. CPython compatibility also means libraries can be developed and tested in CPython too.

I know MicroPython has been doing a lot of asyncio work recently. Do you know if they've switched away from generators and are now CPython compatible? It's been a while since we merged so upstream may have already evolved. If that's the case, we should consider merging in upstream (a large task but this would be another reason to.)

Thanks!

[kvc0]

I know MicroPython has been doing a lot of asyncio work recently. Do you know if they've switched away from generators and are now CPython compatible? It's been a while since we merged so upstream may have already evolved. If that's the case, we should consider merging in upstream (a large task but this would be another reason to.)

It's still backed by yield_from which uses iter and yield.

I don't want to introduce an async API that is incompatible with CPython because then code will be built on top that we'd have to break later. CPython compatibility also means libraries can be developed and tested in CPython too.

Do consider that end user code is written the same way; the coroutine vs. generator thing only concerns library developers. By having the syntax generally available we can gather feedback on concurrency styles that users like, and decouple implementation of coroutine from learning those patterns.
I was pretty disappointed when I found I was unable to write tests for my library code that executed in CPython but even with an asterisk and a construction zone caveat, this is pretty useful functionality to learn.
TBH I'm a little hesitant to build a coroutine implementation when we don't even know how popular concurrent paradigms will be in the CircuitPython community. I.e., if a dozen people want it, we should just use whatever MicroPython does and live with it. If it's generally desirable, it's worth investing in a CPython subset compatibility.
I'll plan to maintain a fork if this isn't a thing you want in CircuitPython. I'm definitely in that dozen!

[tannewt]

I know MicroPython has been doing a lot of asyncio work recently. Do you know if they've switched away from generators and are now CPython compatible? It's been a while since we merged so upstream may have already evolved. If that's the case, we should consider merging in upstream (a large task but this would be another reason to.)

It's still backed by yield_from which uses iter and yield.

Thanks for the links!

I don't want to introduce an async API that is incompatible with CPython because then code will be built on top that we'd have to break later. CPython compatibility also means libraries can be developed and tested in CPython too.

Do consider that end user code is written the same way; the coroutine vs. generator thing only concerns library developers. By having the syntax generally available we can gather feedback on concurrency styles that users like, and decouple implementation of coroutine from learning those patterns.

I understand that but in this case library developers are our users.

I was pretty disappointed when I found I was unable to write tests for my library code that executed in CPython but even with an asterisk and a construction zone caveat, this is pretty useful functionality to learn.

TBH I'm a little hesitant to build a coroutine implementation when we don't even know how popular concurrent paradigms will be in the CircuitPython community. I.e., if a dozen people want it, we should just use whatever MicroPython does and live with it. If it's generally desirable, it's worth investing in a CPython subset compatibility.

How big is the coroutine API? Could we make a generator look like a subset of the CPython coroutine api? If it's not much work, then I think it's worth doing. Lots of thought went into the CPython APIs.

I'll plan to maintain a fork if this isn't a thing you want in CircuitPython. I'm definitely in that dozen!

Thanks! I'd like to have the async and await keywords but I want them to work like CPython.

@dpgeorge @jimmo Any insight why MicroPython's asyncio isn't CPython compatible?

[dpgeorge]

Asyncio is a pretty complex topic and there's been a lot of work done on it in MicroPython over the years. Support for coroutines via generators was done for efficiency and for the most part users will not notice any difference here. Our uasyncio module is largely CPython compatible and I'd suggest you rebase/merge v1.13 of MicroPython (once it's released, should be soon) as just use that as-is.

[tannewt]

@dpgeorge Thanks for the reply! What sort of efficiency? Execution speed or implementation. I'd much prefer a CPython compatible solution.

[dpgeorge]

Mainly efficiency of implementation.

There are lots of layers involved with an asyncio implementation and we've prioritised making the user-facing layers CPython compatible. I don't think it's realistic (or necessary) to make everything CPython compatible, for example it probably requires reference counting to get task clean-up to work the same.

So I guess the question is: what does "CPython compatible" mean to you?

[kvc0]

In making my weak scheduler PoC I saw that the lack of __await__ and send() on the awaitable object produced by an async function invocation were obstacles to compatibility at least insofar as writing unit tests. I think if I did not have to advance my tasks via next() we would be not too far off. I don't think a full CPython coroutine implementation is necessary here to get the overlap good enough that CircuitPython (limited) code works in a CPython (full) runtime.

[kvc0]

To be clear though, it is really just that you can't implement an asyncio-api-compatible pure-python scheduler with this bare generator async/await. If I had written a CircuitPython-specific C module that did task scheduling (reeeeimplement asyncio or uasyncio) there would not be Python unit tests and the user-visible code would be compatible with CPython (modulo the library that would need to be built in blinka).
I consider this gap to be a temporary thing with a low-pain forward migration story for early adopters; but it's okay to keep it a separate fork if it's just not a CircuitPython option to roll it out in stages.

[dpgeorge]

In making my weak scheduler PoC I saw that the lack of __await__ and send() on the awaitable object produced by an async function invocation were obstacles to compatibility at least insofar as writing unit tests.

@WarriorOfWire are you able to provide/link to some code which uses __await__ and send() in a way that works under CPython but not MicroPython? send() should work.

[kvc0]

@dpgeorge I have committed an error; I apologize for the needless sink on your time. Indeed send() works like CPython. Thank you for pointing out my error.

here is the diff that allows the (incredibly) budget scheduler to work under CPython.

Seriously Damian, thank you ❤️.

[dpgeorge]

Awesome!

[kvc0]

@dpgeorge while you're in the neighborhood - how would I implement a suspending function in Python using async/await that works both in CPython and Micropython?

Consider:

async def go_next_time():
  yield

async def print_next_time(message):
  await go_next_time()
  print(message)

co = print_next_time('hello')
co.send(None)
co.send(None)

This works in CircuitPython but not in CPython. It is an abuse of the encapsulation, but I'm not sure what alternative means to trigger a generator suspension would be more appropriate given that I'm limited to using Python for this.

Here's what JetBrains says about it:

[kvc0]

Right, figured out a way that works in both to provide a "sleep":

class TimedWait:
    def __init__(self, duration):
        self.duration = duration
        self.start = time.monotonic()

    def __await__(self):
        yield from self._wait()

    def __iter__(self):
        yield from self._wait()

    def _wait(self):
        while time.monotonic() < self.start + self.duration:
            yield

It's a little circuitous but it will do until we have a C module in place.

[dpgeorge]

@WarriorOfWire you can do __await__ = __iter__ in your class definition, instead of providing 2x identical functions.

[kvc0]

So this bad Python scheduler serves as a proof that there is a subset of functionality that overlaps sufficiently with CPython to write a meaningful pure Python coroutine scheduler (which should not be the ultimate goal here imho).

The async/await syntax of my demo application has been unchanged through this tumult BudgetScheduler has remained API compatible even through misunderstandings. Creating syntax-compatible (CPY-CircuitPy) Python libraries wrapping C device drivers is eminently plausible.

Notwithstanding all of this, it would still make sense to me if @tannewt does not want to take syntax that does not have full vertical C support. I will continue to benefit greatly from my bad scheduler, as it handily beats ad-hoc state machines, but in truth it is not a suitable general purpose substitute for asyncio or uasyncio.

[kvc0]

@WarriorOfWire you can do __await__ = __iter__ in your class definition, instead of providing 2x identical functions.

Yes you're right, this suffices for compatibility:

class _CallMeNextTime:
    def __iter__(self):
        yield

    __await__ = __iter__

[tannewt]

So I guess the question is: what does "CPython compatible" mean to you?

@dpgeorge Thanks for jumping in on this! I really appreciate your input.

My philosophy is that "CPython compatible" means that any code written in CircuitPython that only imports modules available in CPython works without modification in CPython. The corollary of this is that any CircuitPython-only code must import a module not available in CPython. This means that CircuitPython-only code will error consistently early at import time. (It also gives us a hook to provide a python library in CPython to implement the functionality.)

I don't know a lot about asyncio so please correct me if I'm wrong. It seems to me that by having async and await produce generators instead of coroutines we'd risk having folks use them like generators, which wouldn't work in CPython. It feels like we could share the generator parts we need with a new coroutine type similar to how str and bytes share implementation details.

[deshipu]

You can use generators like coroutines even back in Python 2, ever since yield was introduced — in fact the early async proofs of concepts were done that way.

[deshipu]

I think that at some point they actually added a check to async/await that raises an error if you call it with a generator, but that was relatively late in the development.

[tannewt]

Anyone want to do the work to make sure we can't use the returned object differently than we would in CPython? Otherwise I'll close this PR.

[tannewt]

Closing for now. Anyone should feel free to pick this branch up and make a new PR.

My concern as-is is that the returned object should have a Python API that is a subset of the object returned in CPython. I don't care how it is implemented internally.