Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
Browse files

Make DbConnection more generic and move into pyrseas.lib.

 * docs/database.rst: Explain new CatDbConnection class.
 * docs/dbconn.rst: Explain generic usage of extended DbConnection.
   Add new automethods close, commit, rollback, execute and remove
   autoattribute.version.
 * docs/{dbobject,index}.rst: Update links.
 * pyrseas/database.py (CatDbConnection): New class derived from
   DbConnection.  (Database.__init__): Change arguments and initialize
   dbconn here.
 * pyrseas/dbconn.py: Deleted/moved to pyrseas/lib.
 * pyrseas/dbobject/__init__.py (DbObjectDict.fetch): Remove connect
   but add rollback.
 * pyrseas/dbobject/operclass.py (OperatorClassDict._from_catalog):
   Add rollback's after fetchall's.
 * pyrseas/dbobject/table.py (ClassDict._from_catalog): Add rollback
   after fetchall.
 * pyrseas/dbtoyaml.py (main): Initialize Database directly. Get
   password here.
 * pyrseas/lib/__init__.py: New submodule.
 * pyrseas/lib/dbconn.py: Moved from pyrseas. Made more generic so
   that it can be used for other purposes.  Renamed _execute to
   execute, added close, commit and rollback methods.  Don't rollback
   after fetchone and fetchall.  Exclude set search_path calls.
 * pyrseas/testutils.py (PyrseasTestCase.database): Initialize
   Database directly.
 * pyrseas/yamltodb.py (main): Initialize Database directly. Get
   password here. Use DbConnection execute, rollback and commit
   instead of Psycopg.
  • Loading branch information...
commit 6adadd1ec8fa7b84b9a1c9a916a41e82a93d8ac0 1 parent e20732d
@jmafc jmafc authored
View
6 docs/database.rst
@@ -8,8 +8,10 @@ The :mod:`database` module defines :class:`Database`.
Database
--------
-A :class:`Database` is initialized with a
-:class:`~pyrseas.dbconn.DbConnection` object. It consists of one or
+A :class:`Database` is initialized from a
+:class:`~pyrseas.database.CatDbConnection` object (a specialized class
+derived from :class:`~pyrseas.lib.dbconn.DbConnection`).
+It consists of one or
two :class:`Dicts`. A :class:`Dicts` object holds various dictionary
objects derived from :class:`~pyrseas.dbobject.DbObjectDict`, e.g.,
:class:`~pyrseas.dbobject.schema.SchemaDict`,
View
25 docs/dbconn.rst
@@ -1,7 +1,7 @@
Database Connections
====================
-.. module:: pyrseas.dbconn
+.. module:: pyrseas.lib.dbconn
The :mod:`dbconn` module defines :class:`DbConnection`.
@@ -10,8 +10,17 @@ Database Connection
A :class:`DbConnection` is a helper class representing a connection to
a `PostgreSQL <http://www.postgresql.org>`_ database via the `Psycopg
-<http://initd.org/psycopg/>`_ adapter. A :class:`DbConnection` is not
-necessarily connected. It will typically connect to the database when
+<http://initd.org/psycopg/>`_ adapter. It provides an easier
+interface than direct access to Psycopg. For example::
+
+ >>> from pyrseas.lib.dbconn import DbConnection
+ >>> db = DbConnection('dbname')
+ >>> db.fetchone("SHOW server_version")[0]
+ >>> db.commit()
+
+A :class:`DbConnection` is not necessarily connected.
+In the case of Pyrseas :doc:`dbtoyaml </dbtoyaml>` and :doc:`yamltodb
+</yamltodb>`, it will typically connect to the database when
the :class:`~pyrseas.dbobject.DbObjectDict`
:meth:`~pyrseas.dbobject.DbObjectDict.fetch` method is first
invoked. It is normally disconnected just before the
@@ -22,8 +31,14 @@ invoked. It is normally disconnected just before the
.. automethod:: DbConnection.connect
+.. automethod:: DbConnection.close
+
+.. automethod:: DbConnection.commit
+
+.. automethod:: DbConnection.rollback
+
+.. automethod:: DbConnection.execute
+
.. automethod:: DbConnection.fetchone
.. automethod:: DbConnection.fetchall
-
-.. autoattribute:: DbConnection.version
View
2  docs/dbobject.rst
@@ -45,7 +45,7 @@ Database Object Dictionary
A :class:`DbObjectDict` represents a collection of :class:`DbObject`'s
and is derived from the Python built-in type :class:`dict`. If a
-:class:`~pyrseas.dbconn.DbConnection` object is used for
+:class:`~pyrseas.lib.dbconn.DbConnection` object is used for
initialization, an internal method is called to initialize the
dictionary from the database catalogs. The :class:`DbObjectDict`
:meth:`fetch` method fetches all objects using the :attr:`query`
View
2  docs/index.rst
@@ -57,7 +57,7 @@ API Reference
-------------
Currently, the only external APIs are the classes
-:class:`~pyrseas.dbconn.DbConnection` and
+:class:`~pyrseas.lib.dbconn.DbConnection` and
:class:`~pyrseas.database.Database` and the methods
:meth:`~pyrseas.database.Database.to_map` and
:meth:`~pyrseas.database.Database.diff_map` of the latter. Other
View
31 pyrseas/database.py
@@ -10,6 +10,7 @@
system catalogs. The `ndb` Dicts object defines the schemas based
on the `input_map` supplied to the `from_map` method.
"""
+from pyrseas.lib.dbconn import DbConnection
from pyrseas.dbobject.language import LanguageDict
from pyrseas.dbobject.cast import CastDict
from pyrseas.dbobject.schema import SchemaDict
@@ -42,6 +43,26 @@ def flatten(lst):
yield elem
+class CatDbConnection(DbConnection):
+ """A database connection, specialized for querying catalogs"""
+
+ def connect(self):
+ """Connect to the database"""
+ super(CatDbConnection, self).connect()
+ try:
+ self.execute("set search_path to public, pg_catalog")
+ except:
+ self.rollback()
+ self.execute("set search_path to pg_catalog")
+ self.commit()
+ self._version = self.conn.server_version
+
+ @property
+ def version(self):
+ "The server's version number"
+ return self._version
+
+
class Database(object):
"""A database definition, from its catalogs and/or a YAML spec."""
@@ -77,12 +98,16 @@ def __init__(self, dbconn=None):
self.usermaps = UserMappingDict(dbconn)
self.ftables = ForeignTableDict(dbconn)
- def __init__(self, dbconn):
+ def __init__(self, dbname, user=None, pswd=None, host=None, port=None):
"""Initialize the database
- :param dbconn: a DbConnection object
+ :param dbname: database name
+ :param user: user name
+ :param pswd: user password
+ :param host: host name
+ :param port: host port number
"""
- self.dbconn = dbconn
+ self.dbconn = CatDbConnection(dbname, user, pswd, host, port)
self.db = None
def _link_refs(self, db):
View
111 pyrseas/dbconn.py
@@ -1,111 +0,0 @@
-# -*- coding: utf-8 -*-
-"""
- pyrseas.dbconn
- ~~~~~~~~~~~~~~
-
- A `DbConnection` is a helper class representing a connection to a
- PostgreSQL database.
-"""
-import os
-import sys
-import getpass
-
-from psycopg2 import connect
-from psycopg2.extras import DictConnection
-
-
-class DbConnection(object):
- """A database connection, possibly disconnected"""
-
- def __init__(self, dbname, user=None, pswd=False, host=None, port=None):
- """Initialize the connection information
-
- :param dbname: database name
- :param user: user name
- :param pswd: prompt user for password
- :param host: host name
- :param port: host port number
- """
- self.dbname = dbname
- self.user = user
- if pswd:
- self.pswd = " password=%s" % getpass.getpass()
- else:
- self.pswd = ''
- if host is None or host == '127.0.0.1' or host == 'localhost':
- self.host = ''
- else:
- self.host = "host=%s " % host
- if port is None or port == 5432:
- self.port = ''
- else:
- self.port = "port=%d " % port
- self.conn = None
- self._version = 0
-
- def connect(self):
- """Connect to the database
-
- If user is None, the USER environment variable is used
- instead.
- """
- try:
- self.conn = connect("%s%sdbname=%s user=%s%s" % (
- self.host, self.port, self.dbname,
- self.user or os.getenv("USER"), self.pswd),
- connection_factory=DictConnection)
- except Exception as exc:
- if str(exc)[:6] == 'FATAL:':
- sys.exit("Database connection error: %s" % str(exc)[8:])
- else:
- raise
- try:
- self._execute("set search_path to public, pg_catalog")
- except:
- self.conn.rollback()
- self._execute("set search_path to pg_catalog")
- self.conn.commit()
- self._version = int(self.fetchone("SHOW server_version_num")[0])
-
- def _execute(self, query):
- """Create a cursor, execute a query and return the cursor"""
- curs = self.conn.cursor()
- try:
- curs.execute(query)
- except Exception as exc:
- exc.args += (query, )
- raise
- return curs
-
- def fetchone(self, query):
- """Execute a single row SELECT query and return data
-
- :param query: a SELECT query to be executed
- :return: a psycopg2 DictRow
-
- The cursor is closed and a rollback is issued.
- """
- curs = self._execute(query)
- data = curs.fetchone()
- curs.close()
- self.conn.rollback()
- return data
-
- def fetchall(self, query):
- """Execute a SELECT query and return data
-
- :param query: a SELECT query to be executed
- :return: a list of psycopg2 DictRow's
-
- The cursor is closed and a rollback is issued.
- """
- curs = self._execute(query)
- data = curs.fetchall()
- curs.close()
- self.conn.rollback()
- return data
-
- @property
- def version(self):
- "The server's version number"
- return self._version
View
3  pyrseas/dbobject/__init__.py
@@ -317,7 +317,6 @@ def fetch(self):
:return: list of self.cls objects
"""
- if not self.dbconn.conn:
- self.dbconn.connect()
data = self.dbconn.fetchall(self.query)
+ self.dbconn.rollback()
return [self.cls(**dict(row)) for row in data]
View
8 pyrseas/dbobject/operclass.py
@@ -112,12 +112,16 @@ def _from_catalog(self):
if opclass.storage == '-':
del opclass.storage
self[opclass.key()] = OperatorClass(**opclass.__dict__)
- for (sch, opc, idx, strat, oper) in self.dbconn.fetchall(self.opquery):
+ opers = self.dbconn.fetchall(self.opquery)
+ self.dbconn.rollback()
+ for (sch, opc, idx, strat, oper) in opers:
opcls = self[(sch, opc, idx)]
if not hasattr(opcls, 'operators'):
opcls.operators = {}
opcls.operators.update({strat: oper})
- for (sch, opc, idx, supp, func) in self.dbconn.fetchall(self.prquery):
+ funcs = self.dbconn.fetchall(self.prquery)
+ self.dbconn.rollback()
+ for (sch, opc, idx, supp, func) in funcs:
opcls = self[(sch, opc, idx)]
if not hasattr(opcls, 'functions'):
opcls.functions = {}
View
4 pyrseas/dbobject/table.py
@@ -402,7 +402,9 @@ def _from_catalog(self):
inst.get_dependent_table(self.dbconn)
elif kind == 'v':
self[(sch, tbl)] = View(**table.__dict__)
- for (tbl, partbl, num) in self.dbconn.fetchall(self.inhquery):
+ inhtbls = self.dbconn.fetchall(self.inhquery)
+ self.dbconn.rollback()
+ for (tbl, partbl, num) in inhtbls:
(sch, tbl) = split_schema_obj(tbl)
table = self[(sch, tbl)]
if not hasattr(table, 'inherits'):
View
6 pyrseas/dbtoyaml.py
@@ -5,11 +5,11 @@
from __future__ import print_function
import os
import sys
+import getpass
from argparse import ArgumentParser
import yaml
-from pyrseas.dbconn import DbConnection
from pyrseas.database import Database
from pyrseas.cmdargs import parent_parser
@@ -28,8 +28,8 @@ def main(host='localhost', port=5432, schema=None):
schema=schema)
args = parser.parse_args()
- db = Database(DbConnection(args.dbname, args.username, args.password,
- args.host, args.port))
+ pswd = (args.password and getpass.getpass() or '')
+ db = Database(args.dbname, args.username, pswd, args.host, args.port)
dbmap = db.to_map([args.schema], args.tablist)
if args.output:
fd = args.output
View
0  pyrseas/lib/__init__.py
No changes.
View
116 pyrseas/lib/dbconn.py
@@ -0,0 +1,116 @@
+# -*- coding: utf-8 -*-
+"""
+ pyrseas.lib.dbconn
+ ~~~~~~~~~~~~~~~~~~
+
+ A `DbConnection` is a helper class representing a connection to a
+ PostgreSQL database.
+"""
+import sys
+
+from psycopg2 import connect
+from psycopg2.extras import DictConnection
+
+
+class DbConnection(object):
+ """A database connection, possibly disconnected"""
+
+ def __init__(self, dbname, user=None, pswd=None, host=None, port=None):
+ """Initialize the connection information
+
+ :param dbname: database name
+ :param user: user name
+ :param pswd: user password
+ :param host: host name
+ :param port: host port number
+ """
+ self.dbname = dbname
+ if user is None:
+ self.user = ''
+ else:
+ self.user = " user=%s" % user
+ if pswd is None:
+ self.pswd = ''
+ else:
+ self.pswd = " password=%s" % pswd
+ if host is None or host == '127.0.0.1' or host == 'localhost':
+ self.host = ''
+ else:
+ self.host = "host=%s " % host
+ if port is None or port == 5432:
+ self.port = ''
+ else:
+ self.port = "port=%d " % port
+ self.conn = None
+
+ def connect(self):
+ """Connect to the database"""
+ try:
+ self.conn = connect("%s%sdbname=%s%s%s" % (
+ self.host, self.port, self.dbname, self.user, self.pswd),
+ connection_factory=DictConnection)
+ except Exception as exc:
+ if str(exc)[:6] == 'FATAL:':
+ sys.exit("Database connection error: %s" % str(exc)[8:])
+ else:
+ raise exc
+
+ def close(self):
+ """Close the database connection"""
+ if self.conn and not self.conn.closed:
+ self.conn.close()
+ self.conn = None
+
+ def commit(self):
+ """Commit currently open transaction"""
+ self.conn.commit()
+
+ def rollback(self):
+ """Roll back currently open transaction"""
+ self.conn.rollback()
+
+ def execute(self, query, args=None):
+ """Create a cursor, execute a query and return the cursor
+
+ :param query: text of the statement to execute
+ :param args: arguments to query
+ :return: cursor
+ """
+ if self.conn is None:
+ self.connect()
+ curs = self.conn.cursor()
+ try:
+ curs.execute(query, args)
+ except Exception as exc:
+ self.conn.rollback()
+ curs.close()
+ raise exc
+ return curs
+
+ def fetchone(self, query, args=None):
+ """Execute a single row SELECT query and return row
+
+ :param query: a SELECT query to be executed
+ :param args: arguments to query
+ :return: a psycopg2 DictRow
+
+ The cursor is closed.
+ """
+ curs = self.execute(query, args)
+ row = curs.fetchone()
+ curs.close()
+ return row
+
+ def fetchall(self, query, args=None):
+ """Execute a SELECT query and return rows
+
+ :param query: a SELECT query to be executed
+ :param args: arguments to query
+ :return: a list of psycopg2 DictRow's
+
+ The cursor is closed.
+ """
+ curs = self.execute(query, args)
+ rows = curs.fetchall()
+ curs.close()
+ return rows
View
5 pyrseas/testutils.py
@@ -8,7 +8,6 @@
from psycopg2.extras import DictConnection
from psycopg2.extensions import ISOLATION_LEVEL_AUTOCOMMIT
-from pyrseas.dbconn import DbConnection
from pyrseas.database import Database
@@ -327,8 +326,8 @@ def tearDown(self):
def database(self):
"""The Pyrseas Database instance"""
- return Database(DbConnection(self.db.name, self.db.user,
- host=self.db.host, port=self.db.port))
+ return Database(self.db.name, user=self.db.user, host=self.db.host,
+ port=self.db.port)
class DatabaseToMapTestCase(PyrseasTestCase):
View
14 pyrseas/yamltodb.py
@@ -6,11 +6,11 @@
from __future__ import print_function
import os
import sys
+import getpass
from argparse import ArgumentParser, FileType
import yaml
-from pyrseas.dbconn import DbConnection
from pyrseas.database import Database
from pyrseas.cmdargs import parent_parser
@@ -33,9 +33,8 @@ def main(host='localhost', port=5432):
parser.set_defaults(host=host, port=port, username=os.getenv("USER"))
args = parser.parse_args()
- dbconn = DbConnection(args.dbname, args.username, args.password,
- args.host, args.port)
- db = Database(dbconn)
+ pswd = (args.password and getpass.getpass() or '')
+ db = Database(args.dbname, args.username, pswd, args.host, args.port)
inmap = yaml.load(args.spec)
if args.schlist:
kschlist = ['schema ' + sch for sch in args.schlist]
@@ -53,15 +52,14 @@ def main(host='localhost', port=5432):
if args.onetrans or args.update:
print("COMMIT;")
if args.update:
- dbconn.connect()
try:
for stmt in stmts:
- dbconn.conn.cursor().execute(stmt)
+ db.dbconn.execute(stmt)
except:
- dbconn.conn.rollback()
+ db.dbconn.rollback()
raise
else:
- dbconn.conn.commit()
+ db.dbconn.commit()
print("Changes applied", file=sys.stderr)
if args.output:
fd.close()
Please sign in to comment.
Something went wrong with that request. Please try again.