PyBCAdvancedPython

Other Wiki Pages: PyBc, Session01, Session02, Session03, Session04, Session05, Session06, Session07, Session08, Session09, f2py, swig, Cpython, Cython, PyTables, PyTaps, PythonBots, Django, GIS, AdvancedPython, WxPython, standardlib,

Advanced Python

__Your One, True Love__

Python Boot Camp 2010 - Session 9 - January 14

''Presented by Anthony Scopatz''

As you have seen by now, Python is a rather healthy language. But it is the cardamom that makes the lassi! In "Advanced Python" we'll talk about that extra behind-the-scenes spice that turns Python into your one true love. Sometimes these are simply neat syntactical tricks and sometimes they are basic to how the language works. The parts of Python that we'll talk about here are 'advanced' not because they are difficult (in fact, they are easy to learn). Rather, it is because you can usually get by perfectly well without them! However, knowing them makes your life that much easier.

Advanced Python Topics:

1. The Zen of Python 1. Special (Magic) Functions, Methods, & Modules 1. Decorators ( @) 1. The lambda Expression 1. The with Statement 1. Scripting and subprocess

__The Zen of Python__

Alright, let's start this!

#!python import this

>>> The Zen of Python, by Tim Peters
>>> >>> Beautiful is better than ugly.
>>> Explicit is better than implicit.
>>> Simple is better than complex.
>>> Complex is better than complicated.
>>> Flat is better than nested.
>>> Sparse is better than dense.
>>> Readability counts.
>>> Special cases aren't special enough to break the rules.
>>> Although practicality beats purity.
>>> Errors should never pass silently.
>>> Unless explicitly silenced.
>>> In the face of ambiguity, refuse the temptation to guess.
>>> There should be one-- and preferably only one --obvious way to do it.
>>> Although that way may not be obvious at first unless you're Dutch.
>>> Now is better than never.
>>> Although never is often better than *right* now.
>>> If the implementation is hard to explain, it's a bad idea.
>>> If the implementation is easy to explain, it may be a good idea.
>>> Namespaces are one honking great idea -- let's do more of those!

Yes to all of these. The above was written by Tim Peters back in 2004 and can be found at PEP 20 (http://www.python.org/dev/peps/pep-0020/). These mantras are meant to be more guiding principles and less strict coding standards. It is a good idea to keep them around in your back pocket while you write. However, some of them should be more rigidly adhered to than others.

"''__Special cases aren't special enough to break the rules.__''" - In some languages, which we won't mention by name, there exists the basic syntax which is constantly overridden to perform even the simplest of tasks. The core language does an excellent job of avoiding special cases. You should too. In analogy to the real world, special cases are like nearly every commonly used verb in English. But your computer doesn't speak English. It probably speaks something closer to Lojban (http://en.wikipedia.org/wiki/Lojban). Try to talk in Lojban to your computer.

"''__There should be one-- and preferably only one --obvious way to do it.__''" - It is hard for me to stress how central this is to Python. A comparison

between Python's numerical types (http://docs.python.org/library/stdtypes.html#numeric-types-int-float-long-complex) and C-based languages numbers serves to illustrate this point very well.

If you talk to a mathematician (http://www.mathsisfun.com/sets/number-types.html), they'll tell you about all sorts of basic numerical types. Namely there are the Counting, Integer, Rational, Irrational, Algebraic, Imaginary, and Complex numbers. However, many of the number sets are strict subsets of one of the other number types.

The pythonic way of interpreting these mathematical facts is to reduce these to only 4 types! They are int(), float(), long(), and complex(). Moreover unless you really need long() or complex(), you will probably only ever need to use int() and float(). Amazing.

On the other hand, C-languages take the opposite approach and specifies the number types explicitly. For instance, ' int' will still declare an integer but an ' unsigned int' is actually a counting number! Arguably, the functionality of restricting to only non-negative integers is no longer necessary for most modern applications. This is because computer memory is cheap and abundant so storing a (+/-) is a non-issue. Additionally in a worst-case scenario, different C-compilers may specify different precision for different numerical types. This can cause no end of headaches.

The fact that python handles basic numbers in an intuitive, standardized way is one of its central advantages as a language.

__Special (Magic) Functions, Methods, & Modules__

Magic (or 'special' in Python) functions are the language's way of explicitly altering how the language works. Special functions always have the form of two underscores before & after the name ( __*__). They are a way for you to get your classes, functions, methods to do ''exactly'' what you want them to without resorting to ugly kludges. More information can be found at [http://docs.python.org/reference/datamodel.html#special-method-names].

NOTE: Special functions in ''Python'' are __not__ related to the special functions we saw yesterday in ''!SciPy''. The functions present in the ** scipy.special** module are commonly used (though not universally applicable) mathematical functions. Python special functions indicate that this is where the 'magic' happens in the ''language''.

__Special Functions Class Example:__ Let's see how special methods work with regards to classes.

#!CodeExample
#!python

class MyClass: def __init__(self, n, s=""):

self.n = int(n) self.s = str(s)

mc = MyClass(5.0, "Five")

print(mc.n) print(mc.s) print(mc)

>>> 5
>>> Five
>>> <__main__.MyClass instance at 0x7fd977fd25a8>

Here, the special __init__() method acts as a constructor. It is called once when the object is initialized. All the initializer does here is assign an integer n and a string {{{s}}} to itself. Of course, trying to print {{{MyClass}}} itself is not particularly useful. That {{{mc}}} is located in memory at {{{0x7fd977fd25a8}}} doens't matter to most mortals. But what the {{{print()}}} function does is call {{{str(object)}}}, or more precisely, {{{object.__str__()}}}!

#!CodeExample
#!python

class MyClass: def __init__(self, n, s=""):

self.n = int(n) self.s = str(s)

def __str__(self):

return "Maybe {0} = {1}...".format(self.s, self.n)

mc = MyClass(7.0, "Ten") print(mc)

>>> Maybe Ten = 7...

Note that in both ___init__() and __str__(), the keyword {{{self}}} must be passed to the method so that they can access other attributes in the class's scope! (ASIDE: Technically, {{{self}}} is not a keyword in Python, just a convention. However, I have never seen anyone use anything but {{{self}}}! [http://docs.python.org/tutorial/classes.html#random-remarks]) Other basic converters are defined similarly: || __str__(self) || Called to implement str() conversion.|| || __int__(self) || Called to implement int() conversion.|| || __float__(self) || Called to implement float() conversion.|| || __len__(self) || Called to calculate an object's length via len().||

Finally, documenting in Python is easy, fast, and special. Documentation is stored as a string named object.__doc__.

#!CodeExample
#!python

class MyClass: "Hey"

def __init__(self, n, s=""):

"I am initializing MyClass" self.n = int(n) self.s = str(s)

def __str__(self):

return "Maybe {0} = {1}...".format(self.s, self.n)

__doc__ = __doc__ + " Nonny Nonny"

mc = MyClass(7.0, "Ten") print(mc.__doc__) print(mc.__init__.__doc__)

>>> Hey Nonny Nonny
>>> I am initializing MyClass

Here we see two methods for defining a docstring. The first implicitly set __doc__ by placing a string literal directly underneath the class or function definition. Alternatively, you can modify the __doc__ string explicitly.

#!div style="border: 1px solid #d7d7d7; margin: 1em 1.75em; padding: .25em;

overflow: auto;" Hands-on Example

Modify MyClass to include a special length method. This should return an integer that is already intrinsic to the class somehow...

Then use the built-in function len() to retrieve the length of one of your MyClass objects!

__Special Function Numeric Type Example:__ Probably one of the most useful implementations of special functions is that you can override, emulate, replace numerical types and operators for your classes! Once again in the C-universe, this is known as operator overloading. Rather than letting you fundamentally change the operator (+, -, , /, %, *, //), an operator in Python is simply a flag for calling the special function from two objects (__add__(), __sub__(), etc). Say that x and y are two numerical types. The following expressions are all equivalent (though you'd only ever use the first one):

#!CodeExample
#!python

z = x + y z = x.__add__(y) z = y.__radd__(x)

Analogous expressions exist for all operators. You can find a full set of definitions at [http://docs.python.org/reference/datamodel.html#emulating-numeric-types].

For our example here recall that a metric space is simply some set M with a distance function d defined on it (http://mathworld.wolfram.com/MetricSpace.html). Next, consider the most simplistic set we can: M = {0,1} . That is to say that M is binary. Then we need an equally simple d. In words, suppose I existed in this space. Then if I wanted to travel from from me to me (going nowhere) is zero (x = y iff d(x,y)=0). On the other hand if I wanted to go from myself to anywheres else, the distance would be one x!= y iff d(x,y) = 1). Not terribly practical...

But we can effectively reproduce our simple metric space (and number type) using special functions! First, define a simple number type:

#!CodeExample
#!python

class SimpleNum():

"A Simple Number in a Simple Metric Space." def __init__(self, x):

if int(x) = = 0:

self.val = 0

else:

self.val = 1

def __str__(self):

return str(self.val)

def __int__(self):

return int(self.val)

def __float__(self):

return float(self.val)

def __add__(self, y):

if self.val = = 1:

return SimpleNum(1)

elif int(y) = = 0:

return SimpleNum(0)

else:

return SimpleNum(1)

#__add__() could be more concisely written as: #def __add__(self, y): # return SimpleNum(self.val + SimpleNum(y).val) #where __init__() takes care of the details...

def __radd__(self, y):

"Addition is communative! Here, at least..." return self.__add__(y)

def __sub__(self, y):

y = SimpleNum(y) return SimpleNum(self.val - y.val)

def __rsub__(self, y):

"Since we have no negative numbers, subtraction is also communative!" return self.__sub__(self, y)

def __mul__(self, y):

y = SimpleNum(y) return SimpleNum(self.val * y.val)

def __rmul__(self, y):

if isinstance(y, SimpleNum):

return self.__mul__(y)

else:

return NotImplemented

Great! Let's test the different operators, starting with addition.

#!CodeExample
#!python
#Define some SimpleNums

i = SimpleNum(0) j = SimpleNum(37.0) k = SimpleNum(True)

print("i = {0}".format(i)) print("j = {0}".format(j)) print("k = {0}".format(k)) print("")

#Define SimpleNums from additive operations a = i + 0 b = 0 + j c = a + b

print("a = {0}".format(a)) print("b = {0}".format(b)) print("c = {0}".format(c))

>>> i = 0
>>> j = 1
>>> k = 1
>>> >>> a = 0
>>> b = 1
>>> c = 1

And now for subtraction:

#!CodeExample
#!python
#Define SimpleNums from subtraction

p = a - b q = k - (c + 1) r = k - (c - 1)

print("p = {0}".format(p)) print("q = {0}".format(q)) print("r = {0}".format(r))

>>> p = 1
>>> q = 0
>>> r = 1

And finally, for multiplication:

#!CodeExample
#!python
#Define SimpleNums from multiplication

e = j * b f = q * True g = c * 3 print("e = {0}".format(e)) print("f = {0}".format(f)) print("g = {0}".format(g)) print("")

h = 50.0 * k print("h = {0}".format(h))

>>> e = 1
>>> f = 0
>>> g = 1
>>> >>> Traceback (most recent call last):
>>>   File "PyBC_S9_simple_num.py", line 88, in <module>
>>>     h = 50.0 * k
>>> TypeError: unsupported operand type(s) for *: 'float' and 'instance'

Everything was going great! But what is this :: TypeError. This was thrown for :: h because the {{{__rmul__()}}} method specifically disallows right multiplication of a simple number with anything that isn't a simple number! You can specifically disallow operations between various types. For example, while division of a float by an integer usually makes sense, float division by a string rarely does.

In this short example, we have fairly soundly defined our own metric space in Python! Naturally, this can balloon as much as you need it to. You may grab the above example from [attachment:PyBC_S9_simple_num.py].

__Special Modules Example:__ Lastly, when you write a Python Package (a collection of Python Modules) there is one very special module: :: __init__.py. Think of this as analogous to an object's :: __init__() method but for files. However rather than being called when an object is newly instantiated, {{{__init__.py}}} is called when a package is first imported. Say you have the following directory structure:

pkg/

__init__.py mod1.py mod2.py submodules/

__init__.py submod1.py

notinpkg/

unseenmod1.py

In this case, the following expressions would be roughly equivalent:

#!CodeExample
#!python
#Import the package:

import pkg import pkg.__init__

#Import the submodules: import pkg.submodules import pkg.submodules.__init__

Note that since the directory :: pkg/notinpkg/ does not contain an :: __init__.py file, {{{unseenmod1.py}}} will never be imported by Python!

The great thing about :: __init__.py files is that they can be completely empty! They simply need to exist for the package to work properly. Naturally, you can put whatever code you want in them. For more information please refer to [http://docs.python.org/tutorial/modules.html#packages].

#!div style="border: 1px solid #d7d7d7; margin: 1em 1.75em; padding: .25em;

overflow: auto;" Hands-on Example

Make a new directory called ':: specialpkg' and put the :: MyClass and {{{SimpleNum}}} files inside of it. Now write your own {{{__init__.py}}} file that imports everything from {{{MyClass}}} and {{{SimpleNum}}} automatically when you {{{import specialpkg}}}.

__Decorators (@)__

Decorators are the Python version of macros. Or rather they have the same fundamental goal of macros. The job of a decorator is to inject some bit of code non-intrusively into or around another part of the program. However unlike macros (and preprocessor directives) in other languages, Python decorators are written in the same language as the code itself! You would think that this wouldn't be an earth-shattering idea...

Python decorators to a function are placed before the function definition and are specified syntactically with the ':: @'-symbol. The only requirement on decorators is that they be callable (ie have a :: __call__()) method). Therefore any function can be a decorator as can any {{{class}}} where {{{def __call__(self)}}} is defined may also be a decorator. The documentation describes decorators in the most concise fashion (http://docs.python.org/reference/compound_stmts.html#function-definitions):

@f(x)
@g

def h():

... return something

#The above is equivalent to: def h():

... return something

h = f(x)(g(h))

Let's see how decorators work in practice. Let's define a null decorator, which for any function will return ':: None'.

#!CodeExample
#!python

def null(f):

def null_func(*args, **kwargs):

return None

return null_func

def CrazyFunc(j, m, s=""):

return "{0} wants a {1} in its {2}!".format(s, j, m)

@null def NotSoCrazyFunc(j, m, s=""):

return "{0} wants a {1} in its {2}!".format(s, j, m)

print(CrazyFunc("Banana", 3.14, "My Shower")) print(NotSoCrazyFunc("Banana", 3.14, "My Shower"))

>>> My Shower wants a Banana in its 3.14!
>>> None

As you can see, when :: CrazyFunc is decorated by :: null it becomes not quite the wild time it once was. Moreover {{{null}}} is portable so whenever you wish to kill a function's output all you need to do is decorate its definition. The one subtlety here is that {{{null()}}} doesn't return {{{None}}}. Rather it returns a function {{{null_func()}}}(nested within itself) that returns {{{None}}}. Additionally, {{{null}}} takes a function as an argument (the function to be decorated). It is {{{null_func}}} that accepts any arguments and any keyword arguments. These are the parameters that are passed to your original function at runtime.

On a more practical level, let's define some functions and decorators that will either add one to an integer, square it, or do nothing. Also, to avoid nesting functions, we'll use classes as decorators.

#!CodeExample
#!python

class addone_dec:

def __init__(self, f):

self.f = f

def __call__(self, *args, **kwargs):

return self.f(*args, **kwargs) + 1

class square_dec:

def __init__(self, f):

self.f = f

def __call__(self, *args, **kwargs):

return self.f(*args, **kwargs)**2

#Add one, then square @square_dec def addone(x):

return x + 1

#Square x, then add one @addone_dec def square(x):

return x * x

#Do nothing, then add one, then square, #then add another one, then add a final one @addone_dec @addone_dec @square_dec @addone_dec def donothing(x):

return x

print(addone(2)) print(square(2)) print(donothing(2))

>>> 9
>>> 5
>>> 11

More documentation may be found at [http://docs.python.org/reference/compound_stmts.html#function-definitions]. However, a wonderful tutorial on decorators is available at [http://www.artima.com/weblogs/viewpost.jsp?thread=240808]. Lastly, you can grab the above examples at [attachment:PyBC_S9_decorators.py].

#!div style="border: 1px solid #d7d7d7; margin: 1em 1.75em; padding: .25em;

overflow: auto;" Hands-on Example

Write a decorator that prints the value of the function AFTER it has been called. Then simply return the value of the function without altering it further. Lastly apply your new creation to some functions!

__The lambda Expression__

First off, let me say that lambdas are not scary!

You have probably run across them in someone else's code and they may have seemed like magic. A lambda is simply a way of creating an unnamed (anonymous) function. Such a function is restricted to only one expression (so that lambdas always fit on a single line). See, that wasn't so bad...

A bit of history: Python-lambdas were inspired by an equivalent expression in LISP, a functional language. In fact, LISP has been called "''the most intelligent way to misuse a computer''" (http://www.wisdomandwonder.com/link/2941/lisp-is-the-smartest-way-to-misuse-a-computer). It is probably true; few languages can claim to have books that teach you using solely the Socratic Method ([http://www.amazon.com/Little-LISPer-Third-Daniel-Friedman/dp/0023397632/ref= ntt=at=ep=dpt=5 The Little LISPER] & [http://www.amazon.com/gp/product/0262560992/ref = cm=rdp=product The Little Schemer])! Lamdas in programming are simply an implementation of mathematical Lambda Calculus ([http://en.wikipedia.org/wiki/Lambda=calculus]).

For example, say you needed a quick function that simply tested if a number was equal to 20:

#!CodeExample
#!python
#Declare the function

lambda x: x = = 20

>>> <function <lambda> at 0x7fb2797b3140>

#!CodeExample
#!python
#Now call the function

(lambda x: x = = 20)(10)

>>> False

#!CodeExample
#!python
#Assign it to another variable

g = lambda x: x = = 20 g(20.0)

>>> True

Notice how if you were to define g as a normal function, you would have to use 'def', '{{{()}}}', and '{{{return}}}'. All of this is captured by {{{lambda}}} automatically.

#!CodeExample
#!python
#In lambda form

g = lambda x: x = = 20

#In regular form def g(x):

return (x == 20)

Because :

lambdas return functions, and because they are so small, you can define them in-line as an argument to another function! Lets define a function that prints a string before and after a :: lambda filter has been applied.

#!CodeExample
#!python

def str_filter(s, n, f):

return s + " --> " + f(s, n)

str_filter("Da ", 3, lambda s, n: s*n )

>>> Da  --> Da Da Da

#!CodeExample
#!python

str_filter("Sham Wow ", 4, lambda s, n: s[n:] + s[:n])

>>> Sham Wow  -->  Wow Sham

#!CodeExample
#!python

str_filter("NeverOddOrEven", 500, lambda s, n: s[::-1])

>>> NeverOddOrEven --> nevErOddOreveN

#!CodeExample
#!python

str_filter("My love for the Dutch is undying", 4, lambda s, n: "The Flying " + s.split()[n] + "man")

>>> My love for the Dutch is undying --> The Flying Dutchman

#!div style="border: 1px solid #d7d7d7; margin: 1em 1.75em; padding: .25em;

overflow: auto;" Hands-on Example

Using the lambda keyword and the str_filter(...) function above, write your own filter that replaces the middle character of a string {{{s}}} with {{{n}}} uppercase copies of that character.

Lambdas also make rather concise decorators:

#!CodeExample
#!python

addone_dec = lambda f: (lambda x: x + 1)

@addone_dec def f(x):

return x

f(1) >>> 2

More on lambdas can be found in the documentation at [http://docs.python.org/reference/expressions.html#lambda] and [http://docs.python.org/tutorial/controlflow.html#lambda-forms]. You may get the above examples at [attachment:PyBC_S9_lambda.py].

__The with Statement__

As of Python v2.5, we have had the context manager :: with statement. Basically, for an expression :: EXPR, possibly set to the variable {{{VAR}}}, {{{VAR = EXPR.__enter__()}}} is called before executing the subsequent code block. Moreover, {{{EXPR.__exit__()}}} after the code block is exited for any reason, including exceptions! Such a syntax is built to pass exceptions within the code block on up to the next level. Using {{{with}}} will therefore reduce the number of {{{try}}} statements needed withing EXPR's context. Syntactically,

#!python

with EXPR as VAR:

#implicitly perform VAR = EXPR.__enter__() execute BLOCK of code #implicitly perform EXPR.__exit__()

This has been detailed in PEP 343 (http://www.python.org/dev/peps/pep-0343/).

Probably the most common use of with is for file objects. Here, with will ensure that the file is opened and closed properly even in the face of extraneous errors. The following example will open '{{{afile.txt}}}' and place each line's tenth character in a list.

#!CodeExample
#!python

col_ten = [] with open('afile.txt', 'r') as f:

for line in f:

col_ten.append(line[9])

print(col_ten)

Note that the above will fail if any line has less than ten columns in it. But with ensures that the file will be closed BEFORE Python stops the program.

#!div style="border: 1px solid #d7d7d7; margin: 1em 1.75em; padding: .25em;

overflow: auto;" Exercise Left for the Reader

Define your own class using your own __enter__() and __exit__() methods such that it would be compatible with the {{{with}}} statement. Simply make sure to follow the specifications found at [http://docs.python.org/reference/compound_stmts.html#the-with-statement] and [http://docs.python.org/reference/datamodel.html#context-managers].

For example, try making a Year context that contains a Months and {{{Birthdays}}}. Then for each month, have a {{{Year.birthdays("Month")}}} function that prints all the birthdays in that month. The following should print all birthdays for each month:

#!CodeExample
#!python

with Year({'jenny': 'May 30th', 'Dr. Cyclops': 'January 1'}) as y:

for m in y.months:

print(y.birthdays(m) + ' were all born in ' + m + '!')

__Scripting and subprocess__

Python is often called a scripting language. What this means exactly is best left to those more semantically inclined. Definitional disagreements aside, Python can be ''used'' to do what most people consider scripting (though this is not the language's primary objective). Programs that emulate Shell scripts can be written with a minimum of overhead using the ' subprocess' module.

Historically, there have been no fewer than four methods that existed simultaneously to simply open a pipe and execute a command. (Not very Zen!) As of Python v2.6, these are thankfully deprecated. What we now have is a single, robust subprocess.Popen class. However while Popen is hearty, it is more convenient to use the subprocess.call() function which accepts the same arguments as {{{Popen}}}'s initializer. Arguments to the process are given as a list of strings.

#!CodeExample
#!python

import subprocess

#Find all instances of Tofu in our imitation meat recipes subprocess.call(["grep", "-r", "Tofu", "Recipes/FakeMeat/"])

#Popen has the kwarg "shell=False" #But setting shell=True will cause the first string argument #to be executed in the 'usual way'. subprocess.call("grep -r Tofu Recipes/FakeMeat/", shell=True)

Great, now any commands we write in BASH we can easily wrap in Python! However, better than BASH, we get the benefit of Python's wonderful lists, strings, loops, etc. It should be noted that Gentoo's Portage package management system is effectively just the above methodology on steroids.

Still, there are some subtitles about subprocesses that need to be noted. The first is that :: call() does not return the :: stdout string! Rather it returns the return code of the pipe after it was executed. This is the equivalent of the BASH command $?.

#!CodeExample
#!python

import subprocess

#A successful call will return 0 return_code = subprocess.call("ls -l /home/") print(return_code)

>>> 0

#!CodeExample
#!python
#A failed call will return > 0

return_code = subprocess.call("cp nothere.txt") print(return_code)

>>> 1

However, this doesn't mean that you there is no way to obtain the :: stdout and :: stderr strings. To do this though we have to return to the {{{Popen}}} object and specify that output should go to {{{PIPE}}} rather than {{{None}}}. We can then use the pipe object's {{{communicate()}}} method to grab the value of {{{stdout}}} and {{{stderr}}}. For example, let's make a list of the process identification numbers of all awesome programs we are currently running:

#!CodeExample
#!python

import subprocess

sp = subprocess.Popen("ps ux | grep awesome", stdout = subprocess.PIPE, stderr= subprocess.PIPE, shell = True) spout, sperr = sp.communicate()

PIDs = [] for line in spout.split('n'):

PIDs.append(line.split()[1])

print(PIDs)

>>> ['2158', '6299', '6301']

As you can see, the subprocess module is how to make Python to truly interact with your computer! For more information please see [http://docs.python.org/library/subprocess.html].