Skip to content
This repository

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Abstract

The Cython language makes writing C extensions for the Python language
as easy as writing in Python itself, but it currently has some
shortcomings.

I propose to improve Cython's compiler design by developing and
integrating:

 * a new Parser,
 * an Abstract Syntax Tree ,and
 * a Flow Graph

These enhancements will allow:
 * full parsing of all valid Python 2.5 code
 * easier transition to Python 3000
 * simpler support of new features
 * better reasoning about the code for optimizations

This is not a from-scratch implementation. I worked on similar
projects before and many of the proposed ideas have been already
tested in the CPython compiler and in pypy, another open source python
compiler. (I participated in a pypy sprint in 2006 and worked on an
AST peephole optimizer)

By modifying and adapting existing open source code and leveraging
from my experience, I shall accomplish this project in the available
12 weeks.

The ability to maintain Cython in the long term will be significantly
enhanced by these modifications.

For the suggested enhancements see:
 http://wiki.cython.org/enhancements

For the complete application text see:
 http://wiki.cython.org/FabrizioM/soc2008

Discussion

This document describes the complete proposal for this project.

What is the Objective ?
    * Improve Cython's compiler design, enabling the generation of
      code for Python 2.5 syntax.
    * Pave the way for future improvements and optimizations.

What is Cython ? How it is related to Python?
     The Cython language makes writing C extensions for the Python
     language as easy as writing in Python.

     The Cython Compiler generates C code that makes full use of the
     CPython API, making it easy to mix compiled and non-compiled
     code. It does not convert Python code to a pure C program.

What about the proposal ?
     The current Cython implementation has some shortcomings.
     After studying Cython's design, I propose to improve Cython's
     compiler design by developing and integrating:

    * a new Parser
    * an Abstract Syntax Tree
    * a Flow Graph

     These enhancements will allow full parsing of all valid Python
     2.5 code.

    The central idea in my design is to build an Abstract
    Syntax Tree from the parsed input source.
    The concrete benefit of this approach lead the team of Python
    developers to implement an AST for the Python compiler during the 2.4 -> 2.5 transition
    (see http://www.python.org/dev/peps/pep-0339/).

    (More technical details in "How it will be done?" section on
    http://wiki.cython.org/FabrizioM/soc2008)

Why the need of this project ? Why should be the project approved?
    * Cython's long term maintainability will be significantly
      enhanced by the project
    * The project would benefit the python, Cython and Open Source
      community in several ways

What will the accomplished project deliver?
What is in it for the Cython project ?
 The proposed enhancements will allow:
   1. Simpler support of new features
   2. Full parsing of Python 2.5 syntax
   3. Better reasoning about the code for optimizations
   4. Pave the way to support Python 3000 features
   5. Improve performance: many well know optimizing techniques could
      be implemented, boosting extension speed execution

What is in it for the Python community ?

   1. Better usability, Faster prototyping, development and code reuse
   2. Better module support for the different Python API versions
   3. Easier porting for existing C code, giving the python community
      more extensions to use and leverage
   4. Improve performance: existing python modules could be compiled to C

What is in it for the Open Source community ?

  1. Easier way of integrating python with existing C/C++ applications
  2. C Programs could leverage from the power of Python in a faster
     way

What is in it for me?

   1. This work will be the base of my master's thesis on
      "automatic code generation from dynamics language for high
      performance computers"
   2. Gain a better understanding of Cython and Python internals
   3. Improve my skill in writing optimized Python applications
   4. Learn more on Python and C / C++ integration
   5. Have fun with the community of Cython
   6. Be sponsored by my favorite company (Google)

How do you know that your project is feasible?
   During this week I spent time on the IRC Cython channel discussing
   with the Cython community the pros, cons, caveats and pitfalls of
   my proposal.

   With the objective to understand my idea, to explore the entity of
   the project, and obtain an inside understanding of the written
   code, I started modifying the existing code, and wrote some first
   tests.

   At the moment I have a working test in my local repository for a
   small subset of the above ideas. (see:
   http://wiki.Cython.org/enhancements/AutoGenParser/CythonVisitorTest)

   Cython has a robust test suite (~200 tests, ~4176 line of code).
   I shall heavily use it during development.

   To have a concrete idea of the amount of work to be done I wrote
   some documents describing the expected behavior.

   http://wiki.Cython.org/enhancements/Parser
   http://wiki.Cython.org/enhancements/Ast
   http://wiki.Cython.org/enhancements/Ast/Mutator
   http://wiki.Cython.org/enhancements/FlowGraph

What is the Project Estimate ?
   The schedule constraint is 12 Full Weeks.
   Parser: 1 to 3 weeks
   Ast: 2 to 5 weeks
   Flow Graph: 4 to 6+ weeks
   ( This will be refined on the online version )

Why Me ?

    1. I have done this before
    2. I am an experienced Python and C/C++ programmer
    3. I am competent about project's topic
    4. I am passionate

    (These claims are further developed in
    http://wiki.cython.org/FabrizioM/soc2008)

Conclusion:

    By supporting this project, you can save a lot of time and neurons
    for many developers. This means that more developers will use Cython
    to develop their libraries/extensions, leading to more libraries
    for Python, that will increase Python's adoption by companies...

    And what is one of the key measurement of a language's success?
    The number of companies using the language.

    Help me improve further Python and Cython's success.

    Thank you for your interest.

Fabrizio Milo

Note: for further detailed explanation read the detailed Description:
http://wiki.cython.org/FabrizioM/soc2008

References:
 www.cython.org
 www.python.org
 Pyrex: http://www.cosc.canterbury.ac.nz/greg.ewing/python/Pyrex/
 PyPy see : http://codespeak.net/pypy/dist/pypy/doc/home.html
Something went wrong with that request. Please try again.