Skip to content
This repository

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

The Mutator Interface

  • Author: [:FabrizioM: FabrizioM]
  • Status: Prototype/Unpublished

TableOfContents(2)

This document describes the steps involved in implementing the with_stmt using the new Cython AST.

First I describe the Mutator interface, then an example of its usage for the with_stmt

Mutator Interface

each AST Node have a method "mutator" that accepts a visitor as argument. i.e for the Stmt Node representing a list of statements:

#in ast.py
class Stmt(Node):
    #...
    def mutate(self, visitor):
        visitor._mutate_list(self.nodes)
        return visitor.visitStmt(self)
    #...

Stmt allows the visitor object to modify his sub-nodes calling visitor._mutate_list(self.node) Where the _mutate_list is implemented as follows:

#!python
class ASTVisitor(object):
    #...
    def _mutate_list(self, lst):
        i = 0
        while i < len(lst):
            item = lst[i].mutate(self)
            if item is not None:
                lst[i] = item
                i += 1
            else:
                del lst[i]

    #...

Reading the above code is clear that you can cut an entire branch of the tree simply by returning None in an AstVisitor. for example to remove all the Discard Nodes:

class AstVisitorRemoveDiscardNode:
   def visitDiscard(self, node):
       return None

A simple Example

Let's look at a concrete example: the With statement introduced in the Python2.5 Version [http://www.python.org/dev/peps/pep-0343/ PEP0343]

with EXPR as VAR:
     BLOCK

We know from the pep that the above code is equal to:

#!python
# withstmt.txt
mgr = (EXPR)
exit = mgr.__exit__  # Not calling it yet
value = mgr.__enter__()
exc = True
try:
    try:
        VAR = value  # Only if "as VAR" is present
        BLOCK
    except:
        # The exceptional case is handled here
        exc = False
        if not exit(*sys.exc_info()):
            raise
        # The exception is swallowed if exit() returns true
finally:
    # The normal and non-local-goto cases are handled here
    if exc:
        exit(None, None, None)

since we use the same ast nodes of the existing Python 2.5 we can generate the Ast from the above doing:

#!python
>>>import compiler
>>>compiler.parse( open('withstmt.txt').read() )
Module(None,
Stmt([Assign([AssName('mgr', 'OP_ASSIGN')], Name('EXPR')),
Assign([AssName('exit', 'OP_ASSIGN')], Getattr(Name('mgr'), '__exit__')),
Assign([AssName('value', 'OP_ASSIGN')], CallFunc(Getattr(Name('mgr'), '__enter__'), [], None, None)),
Assign([AssName('exc', 'OP_ASSIGN')], Name('True')),
TryFinally(
    Stmt([
        TryExcept(
        Stmt([Assign([AssName('VAR', 'OP_ASSIGN')], Name('value')),
        Discard(Name('BLOCK'))]), [(None, None, Stmt([Assign([AssName('exc', 'OP_ASSIGN')], Name('False')),
    If([(Not(CallFunc(Name('exit'), [], CallFunc(Getattr(Name('sys'), 'exc_info'), [], None, None), None)),
     Stmt([Raise(None, None, None)]))], None)]))], None)]),
Stmt([If([(Name('exc'), Stmt([Discard(CallFunc(Name('exit'), [Name('None'), Name('None'), Name('None')], None, None))]))], None)]))]))

Since all the above is valid code we can copy and paste that tree in our mutator

#!python

class With(Node):
    def __init__(self, expr, vars, body, lineno=None):
        self.expr = expr
        self.vars = vars
        self.body = body
        self.lineno = lineno

#...
class WithMutator(AstVisitor):

    def visitWith(self, node):
        #since With is also a stmt we can return a stmt
        WITH_REMOD = Stmt([Assign([AssName('mgr', 'OP_ASSIGN')], node.expr), # NOTE here we must provide the actual with expr
           ..
           Assign([AssName('value', 'OP_ASSIGN')], CallFunc(Getattr(Name('mgr'), '__enter__'), [], None, None)),
           Assign([AssName('exc', 'OP_ASSIGN')], Name('True')),
           TryFinally(Stmt([TryExcept(

           Stmt([Assign([AssName('VAR', 'OP_ASSIGN')], Name('value')),

           node.body]),  # NOTE substitute block

               [(None, None, Stmt([Assign([AssName('exc', 'OP_ASSIGN')], Name('False')),
           If([(Not(CallFunc(Name('exit'), [], CallFunc(Getattr(Name('sys'), 'exc_info'), [], None, None), None)),
           Stmt([Raise(None, None, None)]))], None)]))], None)]),
           Stmt([If([(Name('exc'), Stmt([Discard(CallFunc(Name('exit'),
           [Name('None'), Name('None'), Name('None')], None, None))]))], None)]))]))

    # .. update linenumber from node.lineno
    # .. other checks
        return WITH_REMOD;  # return the new subtree

and pass it to the ast

#!python
ast.mutate (WithMutator())

now using the esisting Cython Visitor you just do:

#!python
ast.visit(CythonVisitor())

and that's it!

Summary

We have seen the proposed mutator interface for the new Ast, how it works and how we could handle the With Stmt .

Note: the mutator interface and the ast code is largely inspired from the [http://codespeak.net/pypy/dist/pypy/doc/home.html Pypy] project

Comments

Looks good! A couple of minor comments.

CarlWitty: 1) I don't like the _mutate_list interface, because it feels like the list is an internal property of the AST that shouldn't be exposed to the visitor. I would prefer something like

self.nodes = visitor._mutate_list(self.nodes[:])

Of course, in this case, the only effect of my proposed change is to make things a little bit slower, so feel free to ignore me...

[:FabrizioM: FabrizioM] You are right. An internal property of the AST must not be exposed to the visitor. But this is a mutator, has a different contract with the Node interface. Visitor and Mutator shares only the interface.

CarlWitty: 2) A real implementation would need to come up with new names, instead of always using 'value', 'exc', 'exit'. I'm curious how a slightly more realistic example incorporating this requirement would look.

[:FabrizioM: Fabrizio]: Well since the WITH_REMOD is itself an ast tree you can do something like:

#!python
class AstMutatorRenamer(AstVisitor):
     def __init__(self, name_space, *args):
         self.name_space = name_space
         #...
     def rename (self, name):
         while name in self.name_space:
             name = '_' + name
         return name

     def visitName(self, node):
         node.name = self.reaname(node.name)
         return node

class WithMutator(AstVisitor):

    def visitWith(self, node):
        #...
        WITH_REMOD.mutate (AstMutatorRenamer())
        #...
Something went wrong with that request. Please try again.