Lazy evaluation / generators

[degory]

Add support for recursively generating infinite sequences. We could do this via generators with yield, or with list comprehensions, or something else.

Generators would also give us async / await pretty much for free, as the state machine transform needed would be the same for both.

As we don't have any intermediate representation between syntax trees and CIL, and our CIL representation is very primitive and can't really be reflected on, we'd need to apply the state machine transformation at the syntax tree level. However, we could severely restrict where we support yield to make it easier. For example, yield would still be very useful, even if we only allowed it in linear code or in an if (or in case when we add pattern matching), and disallow it in loops and in exception handlers - lots of recursive use cases would not require yield within loops.

The state machine state needs to hold the generator function's state, which is much the same as a lambda's closure except that it needs to mutable.

The state machine method needs two entry points, meaning we need a class to hold it plus (at least) two methods

Whilst I think the C# compiler generates a single method with a switch statement in it, we're free to do what suits us. Another option would be to generate one method per state. The key things is that the code for each state needs expect nothing on the stack, leave nothing on the stack, and be re-entrant provided it's supplied with a distinct state per invocation

We could restrict generators to lambdas, although this would be limiting in the future if we also want to use the same mechanisms to support async / await

We need a syntax to mark that a function is a generator. Use of yield in the body may be sufficient.

If we don't support yield in loops, we'll need to support forwarding the result of recursive calls to the caller as well as yielding individual values. Supporting some kinds of loop might not be too difficult, because we'll be able to jump into the middle of the loop. However, for loops will be awkward as we expect to be able to store the loop variables in temporaries, which would need to be preserved in generator's state.

We might want to treat recursive lambdas used to generate infinite sequences differently to functions or methods that implement iterators