Support async expressions

[BCSharp]

Since a few months, IronPython has support of async functions, but it is fully implemented using the existing functionality of Python generators. This PR brings the support of async expressions directly into the DLR, so that any IRON language (not just IronPython) can easily have async/await. The actual await mechanism is delegated to the Roslyn-generated state machine, or (starting from .NET 11) to the CLR's runtime-async support. This scheme benefits from any optimizations done for Roslyn and/or CoreCLR.

I do have an IronPython implementation locally that uses it and is an alternative of the existing implementation of PEP 492. It is a great way of testing the code in the DLR. I think the added API of the DLR is now stable, perhaps even the implementation; I will see it when the IronPython side gets stable and complete enough, so it is possible some tweaks on the DLR side may still become necessary.

[BCSharp]

The chosen "language" for async support in the DLR is .NET types and interfaces. This is different that the current async support in IronPython, which emulates CPython and uses Python's types. I think that staying close to .NET is true to the spirit to the IRON languages and allows for seamless integration with the rest of the .NET ecosystem. Therefore the resulting type of an async expression in the DLR is simply Task<object?>, rather than some own class or interface that, potentially, would be more aligned with what Python does and expects. Similarly, the whole model of asynchronous behaviour is the .NET model, with its synchronization contex even loop. On one side it makes it easy to both consume and produce async functions in async functions from/for other (likely strongly typed) languages, like C#. On the other hand, supporting some other non-.NET models like Python's asyncio will require some bridging.

---

The proposed async expression reuses the existing generator expression functionality in the DLR to slice the expression at the await points (by turning them to yield points, invisible to the caller). This is a convenience of reuse; both generators and async expressions are basically state machines, or coroutines, so it is possible to factor it out to better named methods/types. I don't know though if this would be useful. It will be interesting to look into async generators that combine both these concepts in one expression body. .NET (but not .NET Framework) has interface IAsyncEnumerable<T> that is meant to represent exactly that. It can be consumed (in C#) by statement async for and overlaps in purpose with PEP 525 in Python.

[slozier]

Apologies if this seems like a superficial review, most expression stuff is beyond me. 😄 Didn't spot anything that seemed obviously wrong.

[slozier]

Would use nameof(Microsoft.Scripting.Runtime.AsyncHelpers.DriveAsync)`. Makes it easier to find usage.

[BCSharp]

Yes. I thought that ObsoleteAttribute would cause it being an error, but it doesn't.

[slozier]

Guess it doesn't matter either way (just noting the difference), but in AsyncExpression you defined DefaultCancellationToken and DefaultCancellationException.

[BCSharp]

Good catch, I didn't notice it. I prefer the reusable statics, since it is slightly better performance at a minuscule startup cost and this is the style many IronPython expressions employ. To keep it DRY and not to incur more startup cost than necessary, I moved it to Utils. Also some more shared stuff, keeping all that internal.

[BCSharp]

Thanks for the review. The real party will begin in IronPython, where I am trying to marry the .NET async push model with Python's async pull model. 😃

I guess I wanted you to comment on the addition to the public API of the DLR here. Are you OK with this? I wanted it to be stable (though it does warrant a minor version increase) and was careful with defining the public additions. I believe they are sufficient for IronPython's needs (despite the model differences), hence they will be stable, but we will know for sure only when the whole asyncio support is properly implemented.

Another thing is that the async generator expressions in the DLR are .NET only. This is because IAsyncEnumerable is not in .NET Framework. This means that async generators in IronPython will be only available on .NET. We could add a BCL package with the necessary interfaces, but I know how you feel about additional dependencies, and I, too, would rather see the whole implementation be more established before considering something radical as another dependency.

Finally, there is still this use of generator expressions as a convenient way to drive async expressions. I wasn't satisfied with this, but with async generator expressions it actually worked out quite well. Also I don't quite know how to have async expressions fully in the push model, short of reimplementing the Roslyn's async state machine, which I for sure will newer do. It becomes more interesting and doable in .NET 11, but we are not there yet. First, .NET 10 has to become EOS, and even then, there is still .NET Framework.

[slozier]

I'm fine with the new public APIs. They seem to follow the same pattern as other Expression stuff. I was unsure about the Utils for construction but I guess this is how the DLR does it. Another thought that came to mind for construction was extension methods on the Expression class (e.g. Expression.Async), but I guess that might lead to conflics if .NET ever decides to add their own first class AsyncExpression.

Edit: As for the IAsyncEnumerable package, we can see how it goes with the .NET implementation before determining if it's worth the extra dependencies on .NET Framework.