Browse files

initial commit

  • Loading branch information...
0 parents commit c49e406539758115c9a1ff0bc45b7e3b40ec8fd7 @zzzeek committed Sep 24, 2012
Showing with 581 additions and 0 deletions.
  1. +10 −0 .gitignore
  2. +9 −0 MANIFEST.in
  3. +109 −0 README.rst
  4. +3 −0 akiban/__init__.py
  5. +35 −0 akiban/api.py
  6. +83 −0 akiban/impl.py
  7. +151 −0 akiban/psycopg2.py
  8. +7 −0 setup.cfg
  9. +36 −0 setup.py
  10. +12 −0 tests/__init__.py
  11. +126 −0 tests/fixtures.py
10 .gitignore
@@ -0,0 +1,10 @@
+*.pyc
+*.swp
+*.orig
+build
+tmp
+dist
+.venv
+test*.py
+akiban.egg-info/
+.coverage
9 MANIFEST.in
@@ -0,0 +1,9 @@
+recursive-include docs *.html *.css *.txt *.js *.jpg *.png *.py Makefile *.rst *.sty
+recursive-include tests *.py *.dat
+recursive-include akiban *.py *.dat
+
+include README* LICENSE CHANGES* test.cfg
+
+prune docs/build/output
+
+
109 README.rst
@@ -0,0 +1,109 @@
+Akiban for Python provides a DBAPI compatibility layer for
+`Akiban Server <http://www.akiban.com/>`_.
+
+Akiban Server is a new database engine that is similar in many ways to
+well known engines like Postgresql and MySQL. However, it introduces
+some new twists on SQL, including the ability to render "nested" result
+sets using plain SQL.
+
+Akiban Server uses a database protocol that is compatible with
+Postgresql. Any `DBAPI <http://www.python.org/dev/peps/pep-0249/>`_
+written for Postgresql can also work with Akiban
+Server directly. What Akiban for Python provides is a wrapper around
+these DBAPIs so that Akiban's "nested" result system can be used
+transparently, meaning any result row can contain columns which themselves
+contain "sub-cursors".
+
+So far, Akiban for Python implements one extension module for
+the `psycopg2 <http://pypi.python.org/pypi/psycopg2/>`_ DBAPI for Postgresql.
+Psycopg2 is the most widely used DBAPI for Postgresql, is extremely
+performant and stable and supports Python 3.
+
+Usage of Akiban for Python is extremely simple. When using psycopg2,
+the plugin is enabled as a **connection factory** for psycopg2::
+
+ >>> from akiban.psycopg2 import Connection
+ >>> import psycopg2
+
+ >>> connection = psycopg2.connect(host="localhost", port=15432,
+ ... connection_factory=Connection)
+
+The connection above is in every way an ordinary psycopg2 connection object.
+It's special behavior becomes apparent when using Akiban's **nested result set**
+capability::
+
+ >>> cursor = connection.cursor()
+ >>> cursor.execute("""
+ ... select customers.customer_id, customers.name,
+ ... (select orders.order_id, orders.order_info,
+ ... (select items.item_id, items.price, items.quantity
+ ... from items
+ ... where items.order_id = orders.order_id and
+ ... orders.customer_id = customers.customer_id) as items
+ ... from orders
+ ... where orders.customer_id = customers.customer_id) as orders
+ ... from customers
+ ... """)
+
+Above, we've selected from a table ``customers``, including a nested
+result set for ``orders``. Within that of ``orders``, we have another
+nested result against ``items``. Inspecting ``cursor.description``, we
+see the three outermost columns represented, all normally except for
+``orders`` which has a special typecode ``NESTED_CURSOR``::
+
+ >>> cursor.description
+ [(u'customer_id', <psycopg2._psycopg.type 'INTEGER' at 0x10060a368>, None, None, None, None, None), (u'name', <psycopg2._psycopg.type 'STRING' at 0x10060a4c8>, None, None, None, None, None), (u'orders', <object object at 0x1002af0c0>, None, None, None, None, None)]
+
+If we fetch the first row, it looks mostly normal except for one column that contains a "nested cursor"::
+
+ >>> row = cursor.fetchone()
+ >>> row
+ (1, 'David McFarlane', <akiban.api.NestedCursor object at 0x10068e050>)
+
+looking at the ``orders`` column, we can see that the value is itself a cursor, with its own ``.description``::
+
+ >>> subcursor = row[2]
+ >>> subcursor.description
+ [(u'order_id', <psycopg2._psycopg.type 'INTEGER' at 0x10060a368>, None, None, None, None, None), (u'order_info', <psycopg2._psycopg.type 'STRING' at 0x10060a4c8>, None, None, None, None, None), (u'items', <object object at 0x1002af0c0>, None, None, None, None, None)]
+
+Fetching a row from this cursor, we see it has its own nested data::
+
+ >>> subrow = subcursor.fetchone()
+ >>> subrow
+ (101, 'apple related', <akiban.api.NestedCursor object at 0x10068e0d0>)
+
+and continuing the process, we can see ``items`` column of this row contains another nested cursor::
+
+ >>> subsubcursor = subrow[2]
+ >>> subsubcursor.description
+ [(u'item_id', <psycopg2._psycopg.type 'INTEGER' at 0x10060a368>, None, None, None, None, None), (u'price', <psycopg2._psycopg.type 'DECIMAL' at 0x10060a418>, None, None, None, None, None), (u'quantity', <psycopg2._psycopg.type 'INTEGER' at 0x10060a368>, None, None, None, None, None)]
+
+We can also access all levels of ".description" in one step from the
+lead result, using the extension ".akiban_description". This is
+basically the same structure as that of ``cursor.description``, except
+it produces 8-tuples, instead of 7-tuples. The eighth member of the
+tuple contains the sub-description, if any::
+
+ >>> cursor.akiban_description
+ [(u'customer_id', <psycopg2._psycopg.type 'INTEGER' at 0x10068a3c0>, None, None, None, None, None, None), (u'name', <psycopg2._psycopg.type 'STRING' at 0x10068a520>, None, None, None, None, None, None), (u'orders', <object object at 0x1002af0c0>, None, None, None, None, None, [(u'order_id', <psycopg2._psycopg.type 'INTEGER' at 0x10068a3c0>, None, None, None, None, None, None), (u'order_info', <psycopg2._psycopg.type 'STRING' at 0x10068a520>, None, None, None, None, None, None), (u'items', <object object at 0x1002af0c0>, None, None, None, None, None, [(u'item_id', <psycopg2._psycopg.type 'INTEGER' at 0x10068a3c0>, None, None, None, None, None, None), (u'price', <psycopg2._psycopg.type 'DECIMAL' at 0x10068a470>, None, None, None, None, None, None), (u'quantity', <psycopg2._psycopg.type 'INTEGER' at 0x10068a3c0>, None, None, None, None, None, None)])])]
+
+All those descriptions are nice, but how do we just get all those rows
+back? We need to recursively descend through the nested cursors.
+The code below illustrates one way to do this::
+
+ from akiban import NESTED_CURSOR
+
+ def printrows(cursor, indent=""):
+ for row in cursor.fetchall():
+ nested = []
+ out = ""
+ for field, col in zip(cursor.description, row):
+ if field[1] == NESTED_CURSOR:
+ nested.append((field[0], col, indent))
+ else:
+ out += " " + str(col)
+ print indent + out
+ for key, values, indent in nested:
+ printrows(values, "%s %s: " % (indent, key))
+
+
3 akiban/__init__.py
@@ -0,0 +1,3 @@
+__version__ = '0.9'
+
+from .api import NESTED_CURSOR
35 akiban/api.py
@@ -0,0 +1,35 @@
+import collections
+
+NESTED_CURSOR = object()
+
+
+class NestedCursor(object):
+
+ def __init__(self, ctx, arraysize, fields, description_factory):
+ self.ctx = ctx
+ self._fields = fields
+ self._description_factory = description_factory
+ self._rows = collections.deque()
+ self.arraysize = arraysize
+
+ @property
+ def description(self):
+ return self._description_factory(self._fields)
+
+ def fetchone(self):
+ if self._rows:
+ return self._rows.popleft()
+ else:
+ return None
+
+ def fetchall(self):
+ r = list(self._rows)
+ self._rows.clear()
+ return r
+
+ def fetchmany(self, size=None):
+ if size is None:
+ size = self.arraysize
+ l = list(self._rows)
+ r, self._rows = l[0:size], collections.deque(l[size:])
+ return r
83 akiban/impl.py
@@ -0,0 +1,83 @@
+import json
+from .api import NestedCursor
+
+json_decoder = json.JSONDecoder()
+
+_NESTED_OID = 5001
+
+class AkibanResultContext(object):
+
+ def gen_description(self, fields): # pragma: no cover
+ raise NotImplementedError()
+
+ def typecast(self, value, oid): # pragma: no cover
+ raise NotImplementedError()
+
+ def __init__(self, cursor, firstrow):
+ self.cursor = cursor
+ self.fields = _fields_from_row(firstrow)
+
+ @property
+ def arraysize(self):
+ return self.cursor.arraysize
+
+def _fields_from_row(row):
+ document = json_decoder.decode(row[0])
+ return _format_fields(document)
+
+def _filter_row(row, ctx):
+ if row is None:
+ return None
+ document = json_decoder.decode(row[0])
+ return _format_row(document, ctx.fields, ctx)
+
+def _create_rowset(document, fields, ctx):
+ return [
+ _format_row(row, fields, ctx)
+ for row in document
+ ]
+
+def _format_row(document, fields, ctx):
+ row = []
+ for field in fields:
+ if field['type_oid'] == _NESTED_OID:
+ value = NestedCursor(
+ ctx,
+ ctx.arraysize,
+ field['akiban.fields'],
+ ctx.gen_description
+ )
+ value._rows.extend(
+ _create_rowset(
+ document[field['name']],
+ field['akiban.fields'],
+ ctx
+ )
+ )
+
+ else:
+ value = ctx.typecast(
+ document[field['name']],
+ field['type_oid']
+ )
+ row.append(value)
+ return tuple(row)
+
+def _format_fields(document):
+ ret = []
+ for attrnum, rec in enumerate(document):
+ newrec = {
+ 'table_oid': None,
+ 'name': rec['name'],
+ 'column_attrnum': attrnum,
+ 'format': None,
+ 'type_modifier': -1,
+ 'type_size': -1
+ }
+ if 'columns' in rec:
+ newrec['type_oid'] = _NESTED_OID
+ newrec['akiban.fields'] = _format_fields(rec['columns'])
+ else:
+ newrec['type_oid'] = rec['oid']
+ ret.append(newrec)
+ return ret
151 akiban/psycopg2.py
@@ -0,0 +1,151 @@
+from __future__ import absolute_import
+
+import psycopg2
+import psycopg2.extensions
+from .impl import _filter_row, _NESTED_OID, AkibanResultContext
+from .api import NESTED_CURSOR
+
+
+class Cursor(psycopg2.extensions.cursor):
+
+ def execute(self, *arg, **kw):
+ ret = self._super().execute(*arg, **kw)
+ self._setup_description()
+ return ret
+
+ def executemany(self, *arg, **kw):
+ ret = self._super().executemany(*arg, **kw)
+ self._setup_description()
+ return ret
+
+ def _super(self):
+ return super(Cursor, self)
+
+ def _setup_description(self):
+ if super(Cursor, self).description:
+ self._akiban_ctx = Psycopg2ResultContext(
+ self, self._super().fetchone()
+ )
+ else:
+ self._akiban_ctx = None
+
+ def fetchone(self):
+ return _filter_row(
+ self._super().fetchone(),
+ self._akiban_ctx
+ )
+
+ def fetchall(self):
+ return [
+ _filter_row(row, self._akiban_ctx)
+ for row in self._super().fetchall()
+ ]
+
+ def fetchmany(self, size=None):
+ return [
+ _filter_row(row, self._akiban_ctx)
+ for row in self._super().fetchmany(size)
+ ]
+
+ @property
+ def akiban_description(self):
+ if self._akiban_ctx:
+ return self._akiban_ctx.akiban_description
+ else:
+ return None
+
+ @property
+ def description(self):
+ # TODO: I'm going on a "convenient" behavior here,
+ # that the ".description" attribute on psycopg2.cursor
+ # acts like a method that we override below.
+ # Would need to confirm that the Python
+ # C API and/or psycopg2 supports this pattern.
+ if self._akiban_ctx:
+ return self._akiban_ctx.description
+ else:
+ return None
+
+
+_psycopg2_adapter_cache = {
+}
+
+class Psycopg2ResultContext(AkibanResultContext):
+
+ def gen_description(self, fields):
+ return [
+ (rec['name'], _psycopg2_type(rec['type_oid']),
+ None, None, None, None, None)
+ for rec in fields
+ ]
+
+ @property
+ def description(self):
+ return self.gen_description(self.fields)
+
+ @property
+ def akiban_description(self):
+ return self.gen_akiban_description(self.fields)
+
+ def gen_akiban_description(self, fields):
+ return [
+ (rec['name'], _psycopg2_type(rec['type_oid']),
+ None, None, None, None, None,
+ self.gen_akiban_description(rec['akiban.fields'])
+ if 'akiban.fields' in rec else None
+ )
+ for rec in fields
+ ]
+
+ def typecast(self, value, oid):
+ try:
+ # return a cached "adpater" for this oid.
+ # for a particular oid that's been seen before,
+ # this is the only codepath.
+ adapter = _psycopg2_adapter_cache[oid]
+ except KeyError:
+ # no "adapter". figure it out. we don't want to be
+ # calling isinstance() on every row so we cache whether or
+ # not psycopg2 returns this particular oid as a string
+ # or not, assuming it will be consistent per oid.
+ if isinstance(value, basestring):
+ adapter = _psycopg2_adapter_cache[oid] = \
+ psycopg2.extensions.string_types[oid]
+ else:
+ adapter = _psycopg2_adapter_cache[oid] = None
+
+ if adapter:
+ # TODO: do we send the oid or the adapter
+ # as the argument here?
+ return adapter(value, adapter)
+ else:
+ return value
+
+
+def _psycopg2_type(type_oid):
+ if type_oid == _NESTED_OID:
+ return NESTED_CURSOR
+ else:
+ return psycopg2.extensions.string_types[type_oid]
+
+class Connection(psycopg2.extensions.connection):
+ def __init__(self, dsn, async=0):
+ super(Connection, self).__init__(dsn, async=async)
+ self.autocommit = True
+ self._super_cursor().execute("set OutputFormat='json_with_meta_data'")
+ self.autocommit = False
+
+
+ def _super_cursor(self, *arg, **kw):
+ return super(Connection, self).cursor(*arg, **kw)
+
+ def cursor(self):
+ return self._super_cursor(cursor_factory=Cursor)
+
+# TODO: need to get per-connection adapters going
+# (or get akiban to recognize bool, easier)
+psycopg2.extensions.register_adapter(
+ bool,
+ lambda value: psycopg2.extensions.AsIs(int(value))
+ )
+
7 setup.cfg
@@ -0,0 +1,7 @@
+[egg_info]
+tag_build = dev
+
+[nosetests]
+cover-package = akiban
+with-coverage = 1
+cover-erase = 1
36 setup.py
@@ -0,0 +1,36 @@
+import os
+import re
+from distutils.core import setup
+
+v = open(os.path.join(os.path.dirname(__file__), 'akiban', '__init__.py'))
+VERSION = re.compile(r".*__version__ = '(.*?)'", re.S).match(v.read()).group(1)
+v.close()
+
+readme = os.path.join(os.path.dirname(__file__), 'README.rst')
+
+
+setup(name='akiban',
+ version=VERSION,
+ description="Akiban for Python",
+ long_description=open(readme).read(),
+ classifiers=[
+ 'Development Status :: 4 - Beta',
+ 'Environment :: Console',
+ 'Intended Audience :: Developers',
+ 'Programming Language :: Python',
+ 'Programming Language :: Python :: 3',
+ 'Programming Language :: Python :: Implementation :: CPython',
+ 'Programming Language :: Python :: Implementation :: PyPy',
+ 'Topic :: Database :: Front-Ends',
+ ],
+ keywords='Akiban',
+ author='Mike Bayer',
+ author_email='mike@zzzcomputing.com',
+ license='MIT',
+ packages=['akiban'],
+ include_package_data=True,
+ tests_require=['nose >= 0.11'],
+ test_suite="nose.collector",
+ zip_safe=False,
+ install_requires=requires
+)
12 tests/__init__.py
@@ -0,0 +1,12 @@
+from functools import update_wrapper
+
+def fails(description):
+ def decorate(fn):
+ def go(*arg, **kw):
+ try:
+ fn(*arg, **kw)
+ assert False, "Test did not fail as expected"
+ except AssertionError:
+ assert True
+ return update_wrapper(go, fn)
+ return decorate
126 tests/fixtures.py
@@ -0,0 +1,126 @@
+def _table_fixture(connection):
+ connection.autocommit = True
+ cursor = connection.cursor()
+
+ cursor.execute("""
+ CREATE TABLE IF NOT EXISTS customers
+ (
+ customer_id INT NOT NULL PRIMARY KEY,
+ rand_id INT,
+ name VARCHAR(20),
+ customer_info VARCHAR(100),
+ birthdate DATE,
+ some_bool BOOLEAN
+ )
+ """)
+
+ cursor.execute("""
+ CREATE TABLE IF NOT EXISTS orders
+ (
+ order_id INT NOT NULL PRIMARY KEY,
+ customer_id INT NOT NULL,
+ order_info VARCHAR(200),
+ order_date DATETIME NOT NULL,
+ some_bool BOOLEAN,
+ GROUPING FOREIGN KEY(customer_id) REFERENCES customers
+ )
+ """)
+
+ cursor.execute("""
+ CREATE TABLE IF NOT EXISTS items
+ (
+ item_id INT NOT NULL PRIMARY KEY,
+ order_id INT NOT NULL,
+ price DECIMAL(10, 2) NOT NULL,
+ quantity INT,
+ GROUPING FOREIGN KEY(order_id) REFERENCES orders
+ )
+ """)
+ cursor.close()
+ connection.autocommit = False
+
+def _data_fixture(connection):
+ cursor = connection.cursor()
+ cursor.executemany(
+ "INSERT INTO customers VALUES (%s, floor(1 + rand() * 100), %s, %s, %s, %s)",
+ [
+ (1, 'David McFarlane', 'Co-Founder and CEO', '1982-07-16', True),
+ (2, 'Ori Herrnstadt', 'Co-Founder and CTO', '1982-07-16', True),
+ (3, 'Tim Wegner', 'VP of Engineering', '1982-07-16', True),
+ (4, 'Jack Orenstein', 'Software Engineer', '1982-07-16', False),
+ (5, 'Peter Beaman', 'Software Engineer', '1982-07-16', False),
+ (6, 'Thomas Jones-Low', 'Software Engineer', '1982-07-16', True),
+ (7, 'Mike McMahon', 'Software Engineer', '1982-07-16', False),
+ (8, 'Padraig O''Sullivan', 'Software Engineer', '1983-12-09', True),
+ (9, 'Yuval Shavit', 'Software Engineer', '1983-07-05', False),
+ (10, 'Nathan Williams', 'Software Engineer', '1984-05-01', True),
+ (11, 'Chris Ernenwein', 'Software Testing Engineer', '1982-07-16', False),
+ ]
+ )
+
+
+ cursor.executemany(
+ "INSERT INTO orders VALUES(%s, %s, %s, %s)",
+ [
+ (101, 1, 'apple related', '2012-09-05 17:24:12'),
+ (102, 1, 'apple related', '2012-09-05 17:24:12'),
+ (103, 1, 'apple related', '2012-09-05 17:24:12'),
+ (104, 2, 'kite', '2012-09-05 17:24:12'),
+ (105, 2, 'surfboard', '2012-09-05 17:24:12'),
+ (106, 2, 'some order info', '2012-09-05 17:24:12'),
+ (107, 3, 'some order info', '2012-09-05 17:24:12'),
+ (108, 3, 'some order info', '2012-09-05 17:24:12'),
+ (109, 3, 'some order info', '2012-09-05 17:24:12'),
+ (110, 4, 'some order info', '2012-09-05 17:24:12'),
+ (111, 4, 'some order info', '2012-09-05 17:24:12'),
+ (112, 4, 'some order info', '2012-09-05 17:24:12'),
+ (113, 5, 'some order info', '2012-09-05 17:24:12'),
+ (114, 5, 'some order info', '2012-09-05 17:24:12'),
+ (115, 5, 'some order info', '2012-09-05 17:24:12'),
+ (116, 6, 'some order info', '2012-09-05 17:24:12'),
+ (117, 6, 'some order info', '2012-09-05 17:24:12'),
+ (118, 6, 'some order info', '2012-09-05 17:24:12'),
+ (119, 7, 'some order info', '2012-09-05 17:24:12'),
+ (120, 7, 'some order info', '2012-09-05 17:24:12'),
+ (121, 7, 'some order info', '2012-09-05 17:24:12'),
+ (122, 8, 'some order info', '2012-09-05 17:24:12'),
+ (123, 8, 'some order info', '2012-09-05 17:24:12'),
+ (124, 8, 'some order info', '2012-09-05 17:24:12'),
+ (125, 9, 'some order info', '2012-09-05 17:24:12'),
+ (126, 9, 'some order info', '2012-09-05 17:24:12'),
+ (127, 9, 'some order info', '2012-09-05 17:24:12'),
+ (128, 10, 'some order info', '2012-09-05 17:24:12'),
+ (129, 10, 'some order info', '2012-09-05 17:24:12'),
+ (130, 10, 'some order info', '2012-09-05 17:24:12'),
+ (131, 11, 'some order info', '2012-09-05 17:24:12'),
+ (132, 11, 'some order info', '2012-09-05 17:24:12'),
+ (133, 11, 'some order info', '2012-09-05 17:24:12'),
+ ])
+
+ cursor.executemany(
+ "INSERT INTO items VALUES (%s, %s, %s, %s)",
+ [
+ (1001, 101, 9.99, 1),
+ (1002, 101, 19.99, 2),
+ (1003, 102, 9.99, 1),
+ (1004, 103, 9.99, 1),
+ (1005, 104, 9.99, 5),
+ (1006, 105, 9.99, 1),
+ (1007, 106, 9.99, 1),
+ (1008, 107, 999.99, 1),
+ (1009, 107, 9.99, 1),
+ (1010, 108, 9.99, 1),
+ (1011, 109, 9.99, 1),
+ ]
+ )
+ cursor.close()
+
+def _drop_tables(connection):
+ tables = ['items', 'orders', 'customers']
+ connection.rollback()
+ connection.autocommit = True
+ for tname in tables:
+ cursor = connection.cursor()
+ cursor.execute("DROP TABLE %s" % tname)
+ connection.autocommit = False
+

0 comments on commit c49e406

Please sign in to comment.