Skip to content
This repository

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

#pragma section-numbers on

~+Cython and Python 3+~

<<TableOfContents(2)>>

Synopsis

This project will improve the support for Python 3 in Cython, and use the new features in Python 3 to improve Cython's user-friendliness in its pure Python mode. This includes several bug fixes, as well as the use of Python 3 function annotation and class decorators to provide type information for Cython's pure Python mode.

Deliverables

  • Fix the Python 3 related bugs on the Cython issue tracker
  • Implementation of function annotation and decorator support in pure Python mode
  • Extend and improve documentation for the pure Python mode

Project Details

Cython already has mature support for Python 3. However, there are still tickets related to Python 3 left on the issue tracker. Some of the issues are related to the new syntax in Python 3, and some others are about the transition from Python 2 to 3. The first goal is to clear up as many of these as possible.

On the other hand, we want to utilize the new Python 3 features in Cython. A promising idea is to make use of the function annotation (PEP 3107) to declare types for function parameters and return values. In addition, we could use class and function decorators to provide further information for Cython. With these enhancements, Cython's pure Python mode would be more convenient to use, and the ability to use more of Python 3's features will attract users already switching to Python 3.

Fix Python 3 related bugs

Regarding Python 3 support, currently Cython has the following minor flaws: (links in parentheses are to existing Cython trac tickets)

  • Need a cython3 command, and corresponding directives for Python 3. Cython needs this to know it is parsing Python 3 code or Python 2 code, as ambiguities exist (e.g., print(), __future__ directives).
  • Support for exception chaining as defined in PEP 3134 (#423)
  • Change of scoping for catching exceptions in Python 3 according to the semantic changes described in PEP 3110.
  • Emulate Python 3 print() for Python versions < 2.6 (#69)
  • Support the 'nonlocal' keyword and the corresponding behavior (#490). The imminent closure support in Cython makes implementing this much more feasible.
  • 'with' statement with multiple managers (#487)
  • Proper scoping for variables in comprehension (#465), which means the loop variables in the comprehension should live in an individual scope rather than leak to the outside scope.
  • from __future__ import division is not respected by C int (#259)
  • support for Ellipsis ('...') (#488)

These issues will be resolved one by one. We will make sure the issue is passing tests and the patch is posted to trac for review before moving to the next one. If necessary, we will rework on a patch to improve according to the comments given on trac.

Besides the above listed flaws, any Python 3 related bugs that emerge during the project will be also be recorded on trac, and fixed as time allows.

Pure Python mode with Python 3 features

Cython supports a pure Python mode (#1) that allows one to use the exact same code with both Python and Cython. Instead of Cython-specific syntax, type information for variables and function is provided in the form of decorators and function calls. For example:

import cython

@cython.locals(foo=cython.int, bar=cython.p_char)
def myfunc(foo, bar):
    another_var = cython.declare(cython.int)
    ...

Now with Python 3 function annotations, myfunc could be declared in a more convenient way:

import cython

def myfunc(foo: cython.int, bar: cython.p_char) -> cython.double:
    another_var = cython.declare(cython.int)
    ...

Note that with this annotation, we could also provide the return type of myfunc. Mixed use of cython.locals, cython.declare and function annotation will be supported.

Other tools may also want to use function annotation for documentation or type checking. To provide interoperability, we propose a convention that uses tuples to put multiple annotation together. For example, if you want to declare a Cython variable and document the function the same time:

def myfunc(foo: (cython.int, "This is foo")) -> (cython.double, "return value of myfunc"):
    ...

Note that currently Cython do nothing with typed return value for normal def function. How Cython should behave in this situation will be discussed later in Cython community.

The second enhancement is to use decorators as an equivalent of the Cython keywords cdef and cpdef for classes and functions. The Cython community has agreed, and decided on @cython.cclass as a class decorator, and @cython.cfunc and @cython.ccall for cdef and cpdef respectively (#2). Note that class decorator is supported from Python 2.6. Thus, this piece of Python code:

import cython

@cython.cfunc
def myfunc():
    ...

@cython.cclass
class Foo:
    n = cython.declare(cython.int)
    @cython.ccall
    def bar():
        ...

Will be equivalent to the following Cython code:

cdef myfunc():
    ...

cdef class Foo:
    cdef int n
    cpdef bar():
        ...

Put them together, we could have a complete example for the features will be implemented:

import cython as C
from cython import cfunc, ccall, cclass

@cfunc
def myfunc(a:C.int) -> (C.double, "return value of myfunc"):
    ...

@cclass
class Foo:
    @cfunc
    @C.locals(a=C.int, b=C.double)
    def bar(self):
        ...

    @ccall
    def bar2(self, x:C.p_char):
        y = C.declare(C.p_char)
        ...

The code is clear, and we no longer even need a .pxd file to annotate the code. In the future, with a complete type-inferencer, the power of Cython could be fully exploited in pure Python!

An important part of the project will be to extend and improve the documentation for Cython's pure Python mode. Currently, the pure Python mode is only described in the Cython wiki (link). In the course of this project, the existing and newly implemented features will be added to the Cython documentation.

Schedule

In the first half of the coding period, we will focus on bug fixes. Patches will be produced and get ready for Cython community to review, and probably some of them will be get merged sooner, as they are useful and important for Cython. In this way, the work will be easy to evaluate during the mid-term review. On the other hand, these patches will touch every part of Cython, thus this work will be very helpful for me to have a complete view of Cython codebase.

The second part is to implement functionality for decorators and function annotation. At first, we will work on the decorators. As this part might be tricky, we will have more time to resolve emerged issues if we do it earlier. Function annotation support is relatively easier.

In the case that all these tasks are finished early, I will help with other pure Python mode-related features of Cython.

The detailed project schedule is listed below.

Clear the issues related to Python 3 (5 weeks)

  • Week 1: * Support nonlocal keyword (#490) * with with multiple managers (#487) * from __future__ import division is not respected by C int type (#259).
  • Week 2: * Add a cython3 command and corresponding directives for Python 3 * Support for exception chaining (#423) * Change scoping for catching exceptions in Python 3
  • Week 3: * Proper scoping for variables in comprehension (#465) * Support for Ellipsis ('...') (#488) * Emulate Python 3 print() for Python <2.6 (#69)
  • Week 4: * Work on pure Python mode documentation. * Calling people for code reviews on patches remaining from week 1-3.
  • Week 5: * Respond to comments on patches and polish them accordingly. * Get as many patches as possible merged (as a milestone for mid-term review).

Enhancement of decorator support (2 weeks)

  • Week 6: * Write tests for decorator support. * Write tests to assure the namespace of the cython module is compatible between pure Python and Cython. * Start coding work for decorator support.
  • Week 7: * Continue coding work for decorator support.

Function annotation support (2 weeks)

  • Week 8: * Write tests for annotation support. * Export the __annotations__ slot via C-API PyFunction_SetAnnotations(). * Code for using annotations to declare types of argument and function return values.
  • Week 9: * Continue coding work for annotation support. * Write documentation for annotation support.

Final clean up (1 week)

Clean up code, add documentation and test cases. Make sure the work is merge-ready.

Personal Information

Cython and Me

I am a graduate student currently studying at the National University of Singapore. I have more than 4 years experience with Python and I really love this language. My experience includes Python 3 related development in a previous GSoC, by helping SWIG and Boost.Python for Python 3 support. Thus I should familiar enough with Python internals to do this project.

I like the idea of Cython, that one can write programs as fast as in C using the lovely Python language. So I'd like to push the compatibility and convenience of using Cython with Python 3. Hopefully one day we can compile every piece of Python code and gain amazing speedups!

To understand the Cython codebase, I started by fixing some easy Cython bugs. Two patches have been produced so far: #3, #4. The code of Cython is very clear and nice for me to understand. It really demonstrates the power of Python.

Contact Info

E-mail: mailto:baihaoyu@gmail.com

IRC: haoyu on freenode.net

Blog: (Will set one up for this project)

Thanks!

Thanks for the encouragement and suggestions from the Cython community, especially Craig Citro, Stefan Behnel, Robert Bradshaw, Dag Sverre Seljebotn and Ondrej Certik!

Reference

<<Anchor(1)>>[1] http://wiki.cython.org/pure

<<Anchor(2)>>[2] http://codespeak.net/pipermail/cython-dev/2008-November/002925.html

<<Anchor(3)>>[3] http://trac.cython.org/cython_trac/ticket/399

<<Anchor(4)>>[4] http://trac.cython.org/cython_trac/ticket/422

Something went wrong with that request. Please try again.