Overview of python-brine

Brine is a Python module that adds support for the "true" pickling of functions. The default behavior of the pickle library is to reference functions by name alone. This presents a significant problem when the function you wish to serialize is either a lambda or not defined at the top level.

The brine module provides a way to pickle the actual underlying code of a function, including any closures, and then restore them again.

For more advanced features, there is the brine.barrel module. A barrel is a dictionary-like interface for brining multiple functions. Barrel's internal brining step is recursive. This allows anonymous functions to work on closures referring to other anonymous functions (eg: mutually recursive lambdas and the like). It also preserves uniqueness, if you happen to add the same function multiple times.

• python-brine documentation
• python-brine on GitHub
• python-brine on PyPI

Using brine

Before we begin with our examples, let's contrive a function to preform the pickle/unpickle dance. We'll refer to this helper throughout the remainder of the section.

from pickle import Pickler, Unpickler
from cStringIO import StringIO

def pickle_unpickle(value):
    buffer = StringIO()
    Pickler(buffer).dump(value)
    buffer = StringIO(buffer.getvalue())
    return Unpickler(buffer).load()

Anonymous or inner functions

Pickle normally refuses to serialize a function that is not defined in the top level. The BrineFunction class wraps a function in a manner that supports pickling, and will actually put the code and cells into the serialization stream.

We can use brine.brine to wrap a FunctionType instance, and brine.unbrine to unwrap it again.

from brine import brine, unbrine

# create a function that wouldn't normally be supported via pickle
myfun = lambda x: ("Why hello there, %s" % str(x))
myfun("Godzilla") # ==> "Why hello there, Godzilla"

# if we tried this without the brine/unbrine wrapping, we'd get a
# pickle.PicklingError raised all up in our biz
myfun_redux = unbrine(pickle_unpickle(brine(myfun)))

# this is now a copy of the original
myfun_redux("Mothra") # ==> "Why hello there, Mothra"

Here is something more complex -- two functions sharing a captured value (a closure).

def make_actor(line):
    who = ["nobody special"]
    def notice(monster):
        who[0] = str(monster)
    def alert():
        return line % who[0]
    return notice, alert

actor = make_actor("Look out, it's %s!")
notice, alert = actor

notice("Godzilla")
alert() # ==> "Look out, it's Godzilla!"

# duplicate our highly trained actor
actor_redux = unbrine(pickle_unpickle(brine(actor)))
notice_redux, alert_redux = actor_redux

# our copy of the actor functions come out sharing their own new
# closure cell
alert_redux() # ==> "Look out, it's Godzilla!"
notice_redux("Mothra")
alert_redux() # ==> "Look out, it's Mothra!"

Bound instance methods

Pickle normally refuses to serialize bound instance methods. This is somewhat odd, because it can be done by name. The BrineMethod class can be used to wrap a bound instance or class method. Note that because a bound method needs to be associated with an object, that instance will also need to support pickling (and hence will need its class definition available at the top level).

BrineMethod is entirely name-based -- it doesn't try to pickle underlying class code.

# setup a simple class for us to work over
class Obj(object):
    def __init__(self, value=None):
        self.value = value
    def get_value(self):
        return self.value
    def set_value(self, value):
        self.value = value

inst = Obj("Tacos")
getter = inst.get_value
setter = inst.set_value

setter("Carrots")
getter() # ==> "Carrots"

# a little dance to brine and unbrine both bound methods
tmp = (getter, setter)
tmp = unbrine(pickle_unpickle(brine(tmp)))
n_getter, n_setter = tmp

n_getter() # ==> "Carrots"
n_setter("Sandwich")
n_getter() # ==> "Sandwich"

# the original is unaffected
getter() # ==> "Carrots"

Requirements

• Python 2.6 or later (no support for Python 3, the underlying function fields differ a bit)

In addition, following tools are used in building, testing, or generating documentation from the project sources.

• Setuptools
• Coverage.py
• GNU Make
• Pandoc
• Sphinx

These are all available in most linux distributions (eg. Fedora), and for OSX via MacPorts.

Building

This module uses setuptools, so simply run the following to build the project.

python setup.py build

Testing

Tests are written as unittest test cases. If you'd like to run the tests, simply invoke:

python setup.py test

You may check code coverage via coverage.py, invoked as:

# generates coverage data in .coverage
coverage run --source=brine setup.py test

# creates an html report from the above in htmlcov/index.html
coverage html

I've setup travis-ci and coveralls.io for this project, so tests are run automatically, and coverage is computed then. Results are available online:

• python-brine on Travis-CI
• python-brine on Coveralls.io

Documentation

Documentation is built using Sphinx. Invoking the following will produce HTML documentation in the docs/_build/html directory.

cd docs
make html

Note that you will need the following installed to successfully build the documentation:

Documentation is also available online.

Future Enhancements

Some posibile enhancements for future minor versions

• Should we provide a wrapper for exceptions and/or stack traces?
• Should we allow users to extend BrineObject, in the same manner that pickle can be (somewhat) extended today?
• Perhaps a PKI signing step since we are in-fact sending executable code around? This might be better relegated to a separate project.

Author

Christopher O'Brien obriencj@gmail.com

If this project interests you, you can read about more of my hacks and ideas on on my blog.

License

This library is free software; you can redistribute it and/or modify it under the terms of the GNU Lesser General Public License as published by the Free Software Foundation; either version 3 of the License, or (at your option) any later version.

This library is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public License for more details.

You should have received a copy of the GNU Lesser General Public License along with this library; if not, see http://www.gnu.org/licenses/.