Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

Adding Postgres support to database.py #419

Closed
wants to merge 1 commit into from

3 participants

JonChu Ben Darnell Jon Parise
JonChu

This commit will allow for a more robust database.py that covers both major open source sql implementations.

A Connection object has been put into database.py with a method create()
that takes in a constant denoting if you want a Postgres connection or a
MySql one. The Postgres library models that of the MySql library with
minor modifications where necessary due to differences in the databases.

The differences are documented in the code, but two to note are that
Postgres by default no longer supports returning lastrowid, so
execute/executemany now returns None or the result of the query (if
RETURNING is specified).

Postgres also does not have max idle times by default so this was
accounted for in code.

tornado/database.py
((6 lines not shown))
import copy
import MySQLdb.constants
import MySQLdb.converters
import MySQLdb.cursors
+import psycopg2
Jon Parise
jparise added a note

I think you should protect the MySQLdb and psycopg2 imports so that neither is required until the point that the user selects a particular database connection type.

JonChu
JonChu added a note

Yea, I agree. I'll go ahead and do this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
tornado/database.py
((19 lines not shown))
class Connection(object):
+ @staticmethod
+ def create(host, database, db_type, user=None, password=None,
Jon Parise
jparise added a note

Instead of using enumeration-ish constants for db_type, what if you either passed in a string (containing the letters "mysql" or "postgresql") or the desired connection class object (MysqlConnection or PostgresConnection)?

JonChu
JonChu added a note

Passing in strings seems really ugly to me. There's a reason why enums exist, and when code gets a lot larger, strings become unmaintainable and generally exhibit bad code smell.

I like the idea of passing in the class object itself, but I'll need to ponder this a little more before I go ahead and do that.

Jon Parise
jparise added a note

I suppose I don't really see the practical benefit of:

from database import MYSQL, Connection
c = Connection('localhost', 'database', MYSQL)

over:

from database import Connection
c = Connection('localhost', 'database', 'mysql')

I'm still typing the letters "mysql", and both cases will result in some sort of runtime failure if I mispel it. If that MYSQL symbol wrapped a particularly interesting value (other than just a faux enum) and was used in more than one place (other than the create() factory method), I'd consider it more useful.

My opinion here is rather Python-specific, by the way, because it doesn't actually have enumerations in the strict typing sense.

JonChu
JonChu added a note

I decided the right thing to do was remove the static method and factory class altogether and just allow the programmer to use duck typing. :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
JonChu JonChu Adding Postgres support to database.py
database.py has been modified to have a MySQLConnection and PostgresConnection
class, creating connections for each database respectively. The Postgres
library models that of the MySql library with minor modifications where
necessary due to differences in the databases.

The differences are documented in the code, but two to note are that
Postgres by default no longer supports returning lastrowid, so
execute/executemany now returns None or the result of the query (if
RETURNING is specified).

Postgres also does not have max idle times by default so this was
accounted for in code.
9e6f590
Ben Darnell
Owner

I don't really want to add more one-offs to tornado.database, especially if they hardly share any code. I'd consider a refactoring to separate the mysqlisms from the common code so it takes less than 200 lines to add support for a new database, though. I think it would be ideal if there were a tornado.database.BaseConnection class (since the mysql-specific version has unfortunately claimed the name "Connection") which could take either a DB-API connection or a factory function for such a connection. The base class could provide as much functionality as it could from the base API, with database-specific subclasses to provide additional optimizations like MySQL's SSCursors.

Ben Darnell
Owner

The tornado.database module has been removed and now lives at https://github.com/bdarnell/torndb

Ben Darnell bdarnell closed this
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on Dec 14, 2011
  1. JonChu

    Adding Postgres support to database.py

    JonChu authored
    database.py has been modified to have a MySQLConnection and PostgresConnection
    class, creating connections for each database respectively. The Postgres
    library models that of the MySql library with minor modifications where
    necessary due to differences in the databases.
    
    The differences are documented in the code, but two to note are that
    Postgres by default no longer supports returning lastrowid, so
    execute/executemany now returns None or the result of the query (if
    RETURNING is specified).
    
    Postgres also does not have max idle times by default so this was
    accounted for in code.
This page is out of date. Refresh to see the latest.
Showing with 218 additions and 27 deletions.
  1. +218 −27 tornado/database.py
245 tornado/database.py
View
@@ -14,17 +14,14 @@
# License for the specific language governing permissions and limitations
# under the License.
-"""A lightweight wrapper around MySQLdb."""
+"""Factory method for returning a database connection api for either MySQLdb or psycopg2"""
import copy
-import MySQLdb.constants
-import MySQLdb.converters
-import MySQLdb.cursors
import itertools
import logging
import time
-class Connection(object):
+class MySQLConnection(object):
"""A lightweight wrapper around MySQLdb DB-API connections.
The main value we provide is wrapping rows in a dict/object so that
@@ -40,12 +37,29 @@ class Connection(object):
We explicitly set the timezone to UTC and the character encoding to
UTF-8 on all connections to avoid time zone and encoding errors.
"""
+ import MySQLdb
+ import MySQLdb.constants
+ import MySQLdb.converters
+ import MySQLdb.cursors
+
def __init__(self, host, database, user=None, password=None,
- max_idle_time=7*3600):
+ max_idle_time=7*3600):
self.host = host
self.database = database
self.max_idle_time = max_idle_time
+ # Fix the access conversions to properly recognize unicode/binary
+ FIELD_TYPE = self.MySQLdb.constants.FIELD_TYPE
+ FLAG = self.MySQLdb.constants.FLAG
+ CONVERSIONS = copy.copy(self.MySQLdb.converters.conversions)
+
+ field_types = [FIELD_TYPE.BLOB, FIELD_TYPE.STRING, FIELD_TYPE.VAR_STRING]
+ if 'VARCHAR' in vars(FIELD_TYPE):
+ field_types.append(FIELD_TYPE.VARCHAR)
+
+ for field_type in field_types:
+ CONVERSIONS[field_type] = [(FLAG.BINARY, str)] + CONVERSIONS[field_type]
+
args = dict(conv=CONVERSIONS, use_unicode=True, charset="utf8",
db=database, init_command='SET time_zone = "+0:00"',
sql_mode="TRADITIONAL")
@@ -88,13 +102,13 @@ def close(self):
def reconnect(self):
"""Closes the existing database connection and re-opens it."""
self.close()
- self._db = MySQLdb.connect(**self._db_args)
+ self._db = self.MySQLdb.connect(**self._db_args)
self._db.autocommit(True)
def iter(self, query, *parameters):
"""Returns an iterator for the given query and parameters."""
self._ensure_connected()
- cursor = MySQLdb.cursors.SSCursor(self._db)
+ cursor = self.MySQLdb.cursors.SSCursor(self._db)
try:
self._execute(cursor, query, parameters)
column_names = [d[0] for d in cursor.description]
@@ -196,34 +210,211 @@ def _cursor(self):
def _execute(self, cursor, query, parameters):
try:
return cursor.execute(query, parameters)
- except OperationalError:
+ except self.MySQLdb.OperationalError:
logging.error("Error connecting to MySQL on %s", self.host)
self.close()
raise
-class Row(dict):
- """A dict that allows for object-like property access syntax."""
- def __getattr__(self, name):
+class PostgresConnection(object):
+ """A lightweight wrapper around psycopg2 DB-API connections.
+
+ The main value we provide is wrapping rows in a dict/object so that
+ columns can be accessed by name. Typical usage::
+
+ db = database.Connection("localhost", "mydatabase")
+ for article in db.query("SELECT * FROM articles"):
+ print article.title
+
+ Cursors are hidden by the implementation, but other than that, the methods
+ are very similar to the DB-API.
+ """
+ import psycopg2
+ import psycopg2.extensions
+ # Default name for the server side cursor
+ SSCURSOR_NAME = 'sscursor'
+
+ def __init__(self, host, database, user=None, password=None,
+ max_idle_time=None):
+ self.host = host
+ self.database = database
+ self.max_idle_time = max_idle_time
+
+ # Fix the access conversions to properly recognize unicode
+ self.psycopg2.extensions.register_type(self.psycopg2.extensions.UNICODE)
+ self.psycopg2.extensions.register_type(self.psycopg2.extensions.UNICODEARRAY)
+
+ args = dict(host=host, database=database)
+ if user is not None:
+ args["user"] = user
+ if password is not None:
+ args["password"] = password
+
+ self.socket = None
+ pair = host.split(":")
+ if len(pair) == 2:
+ args["host"] = pair[0]
+ args["port"] = int(pair[1])
+ else:
+ args["host"] = host
+ args["port"] = 5432
+
+ self._db = None
+ self._db_args = args
+ self._last_user_time = time.time()
+
try:
- return self[name]
- except KeyError:
- raise AttributeError(name)
+ self.reconnect()
+ except Exception:
+ logging.error("Cannot connect to PostgreSQL on %s",
+ self.host,
+ exc_info=True)
+
+ def __del__(self):
+ self.close()
+
+ def close(self):
+ """Closes this database connection."""
+ if getattr(self, "_db", None) is not None:
+ self._db.close()
+ self._db = None
+
+ def reconnect(self):
+ """Closes the existing database connection and re-opens it."""
+ self.close()
+ self._db = self.psycopg2.connect(**self._db_args)
+
+ def iter(self, query, *parameters):
+ """Returns an iterator for the given query and parameters."""
+ # TODO: Potentially in the future we could allow for asynchronous iteration
+ # because postgres allows for named cursors
+
+ self._ensure_connected()
+ cursor = self._cursor(self.SSCURSOR_NAME)
+ try:
+ # named cursors need to fetch a row before cursor.description is populated
+ self._execute(cursor, query, parameters)
+ first_row = cursor.fetchone()
+ column_names = [d[0] for d in cursor.description]
+ for row in itertools.chain([first_row], cursor):
+ yield Row(zip(column_names, row))
+ finally:
+ cursor.close()
+
+ def query(self, query, *parameters):
+ """Returns a row list for the given query and parameters."""
+ cursor = self._cursor()
+ try:
+ self._execute(cursor, query, parameters)
+ column_names = [d[0] for d in cursor.description]
+ return [Row(itertools.izip(column_names, row)) for row in cursor]
+ finally:
+ cursor.close()
+
+ def get(self, query, *parameters):
+ """Returns the first row returned for the given query."""
+ rows = self.query(query, *parameters)
+ if not rows:
+ return None
+ elif len(rows) > 1:
+ raise Exception("Multiple rows returned for Database.get() query")
+ else:
+ return rows[0]
+ def execute(self, query, *parameters):
+ """Executes the given query, returning the result from the query (e.g.
+ if RETURNING was used) or None."""
+ cursor = self._cursor()
+ try:
+ self._execute_autocommit(cursor, query, parameters)
+ return cursor.fetchone()[0]
+ except self.psycopg2.ProgrammingError:
+ return None
+ finally:
+ cursor.close()
-# Fix the access conversions to properly recognize unicode/binary
-FIELD_TYPE = MySQLdb.constants.FIELD_TYPE
-FLAG = MySQLdb.constants.FLAG
-CONVERSIONS = copy.copy(MySQLdb.converters.conversions)
+ def execute_rowcount(self, query, *parameters):
+ """Executes the given query, returning the rowcount from the query."""
+ cursor = self._cursor()
+ try:
+ self._execute_autocommit(cursor, query, parameters)
+ return cursor.rowcount
+ finally:
+ cursor.close()
-field_types = [FIELD_TYPE.BLOB, FIELD_TYPE.STRING, FIELD_TYPE.VAR_STRING]
-if 'VARCHAR' in vars(FIELD_TYPE):
- field_types.append(FIELD_TYPE.VARCHAR)
+ def executemany(self, query, parameters):
+ """Executes the given query against all the given param sequences.
-for field_type in field_types:
- CONVERSIONS[field_type] = [(FLAG.BINARY, str)] + CONVERSIONS[field_type]
+ We return the result from the query (e.g. if RETURNING was used) or None.
+ """
+ cursor = self._cursor()
+ try:
+ self._executemany_autocommit(cursor, query, parameters)
+ return cursor.fetchone()[0]
+ except self.psycopg2.ProgrammingError:
+ return None
+ finally:
+ cursor.close()
+ def executemany_rowcount(self, query, parameters):
+ """Executes the given query against all the given param sequences.
-# Alias some common MySQL exceptions
-IntegrityError = MySQLdb.IntegrityError
-OperationalError = MySQLdb.OperationalError
+ We return the rowcount from the query.
+ """
+ cursor = self._cursor()
+ try:
+ self._executemany_autocommit(cursor, query, parameters)
+ return cursor.rowcount
+ finally:
+ cursor.close()
+
+ def _ensure_connected(self):
+ # By default PostgreSQL does not close connections that idle
+ if self.max_idle_time is None:
+ self._ensure_connected = self._ensure_connected_without_idle_limit
+ else:
+ self._ensure_connected = self._ensure_connected_with_idle_limit
+ self._ensure_connected()
+
+ def _ensure_connected_with_idle_limit(self):
+ if (self._db is None or
+ (time.time() - self._last_use_time > self.max_idle_time)):
+ self.reconnect()
+ self._last_use_time = time.time()
+
+ def _ensure_connected_without_idle_limit(self):
+ if self._db is None:
+ self.reconnect()
+
+ def _cursor(self, name=None):
+ self._ensure_connected()
+ return self._db.cursor() if name is None else self._db.cursor(name)
+
+ def _execute(self, cursor, query, parameters):
+ try:
+ return cursor.execute(query, parameters)
+ except self.psycopg2.OperationalError:
+ logging.error("Error connecting to PostgreSQL on %s", self.host)
+ self.close()
+ raise
+
+ def _execute_autocommit(self, cursor, query, parameters):
+ try:
+ self._execute(cursor, query, parameters)
+ finally:
+ self._db.commit()
+
+ def _executemany_autocommit(self, cursor, query, parameters):
+ try:
+ return cursor.executemany(query, parameters)
+ finally:
+ self._db.commit()
+
+
+class Row(dict):
+ """A dict that allows for object-like property access syntax."""
+ def __getattr__(self, name):
+ try:
+ return self[name]
+ except KeyError:
+ raise AttributeError(name)
Something went wrong with that request. Please try again.