Author: Robert Bradshaw Status: Draft, some code
The most glaring deficiencies of Cython (in terms of compiling the entire Python grammar) is the lack of closures. This needs to be implemented to support the lambda keyword, generators, nested def statements, and inner classes.
In Python the current execution stack and instruction pointer (into the python byte code) can be extracted and passed around, with execution resuming at any point. I think that this will be unnecessarily complicated, error prone, and likely platform-dependent to do for arbitrary C code.
Instead, I think a new class should be automatically created that represents the current state of a function (local variables and point of execution). There are a fixed number (compile-time-determined) places that the state would need to be set/restored. This would make it easy to do yield statements, and functional closures could access the "outer" scope through an instance of this class.
I started working on generator support during [http://wiki.sagemath.org/days5 Sage days 5] but never found the time to finish it. Since then I understand the issues and scope of this proposal much better.
dagss: I have a basic prototype working for this (inner functions only, not generators); however it can only store Python objects and not typed values in the closure. It will remain like that until CEP 905 is done; and so I won't work on finishing the implementation until after that (but if anyone are interested I can give them the prototype). (The current prototype works by binding methods to a tuple; instead, they should be bound to a cdef class which can have typed members. Also the prototype attempts to figure out used local and external symbols itself; but it should attempt use the Scope objects instead.)