Merge pull request #158 from zodb/mysql-connector

Support for MySQL Connector/Python

.travis.yml

@@ -16,6 +16,15 @@ addons:
 
 matrix:
   include:
+    - python: 2.7
+      env:
+        - ENV=cmysqlconnector
+        - RS_MY_DRIVER="C MySQL Connector/Python"
+    - python: 3.6
+      env: ENV=mysqlconnector
+    - python: pypy-5.4.1
+      env: ENV=mysqlconnector
+
     - python: 3.6
       env: ENV=mysql
     - python: 3.5
@@ -25,8 +34,6 @@ matrix:
     - python: pypy-5.4.1
       env: ENV=mysql
 
-    - python: 3.6
-      env: ENV=pymysql
     - python: 3.5
       env: ENV=pymysql
     - python: 2.7

.travis/setup-cmysqlconnector.sh

@@ -0,0 +1,7 @@
+wget https://dev.mysql.com/get/Downloads/Connector-Python/mysql-connector-python-2.1.5.tar.gz
+tar -xf mysql-connector-python-2.1.5.tar.gz
+cd ./mysql-connector-python-2.1.5
+python ./setup.py install --with-mysql-capi=/usr
+cd ..
+python -c 'import relstorage.adapters.mysql.drivers as D; print(D.preferred_driver_name,D.driver_map)'
+`dirname $0`/mysql.sh

.travis/setup-mysqlconnector.sh

@@ -0,0 +1,3 @@
+pip install https://dev.mysql.com/get/Downloads/Connector-Python/mysql-connector-python-2.1.5.tar.gz
+python -c 'import relstorage.adapters.mysql.drivers as D; print(D.preferred_driver_name,D.driver_map)'
+`dirname $0`/mysql.sh

.travis/setup-umysqldb.sh

@@ -1,5 +1,5 @@
 pip install -U pymysql
 pip install git+https://github.com/NextThought/umysqldb.git#egg=umysqldb
 export RS_MY_DRIVER=umysqldb
-python -c 'import relstorage.adapters._mysql_drivers as D; print(D.preferred_driver_name,D.driver_map)'
+python -c 'import relstorage.adapters.mysql.drivers as D; print(D.preferred_driver_name,D.driver_map)'
 `dirname $0`/mysql.sh

CHANGES.rst

@@ -15,12 +15,15 @@
   done through the use of prepared statements for the most important
   queries and the new `'ON CONFLICT UPDATE'
   <https://wiki.postgresql.org/wiki/What's_new_in_PostgreSQL_9.5#INSERT_..._ON_CONFLICT_DO_NOTHING.2FUPDATE_.28.22UPSERT.22.29>`_
-  syntax.
+  syntax. See :pr:`157` and :issue:`156`.
 - The umysqldb driver no longer attempts to automatically reconnect on
   a closed cursor exception. That fails now that prepared statements
   are in use. Instead, it translates the internal exception to one
   that the higher layers of RelStorage recognize as requiring
   reconnection at consistent times (transaction boundaries).
+- Add initial support for the `MySQL Connector/Python
+  <https://dev.mysql.com/doc/connector-python/en/>`_ driver. See
+  :issue:`155`.
 
 2.0.0 (2016-12-23)
 ==================

doc/db-specific-options.rst

@@ -30,11 +30,22 @@ PostgreSQL Adapter Options
 The PostgreSQL adapter accepts:
 
 driver
-    Either "psycopg2" or "psycopg2cffi" for the native libpg based
-    drivers. "pg8000" is a pure-python driver suitable for use with
-    gevent.
+    The possible options are:
 
-    .. note:: pg8000 requires PostgreSQL 9.4 or above for BLOB support.
+    psycopg2
+      A C-based driver that requires the PostgreSQL development
+      libraries. Optimal on CPython, but not compatible with gevent.
+
+    psycopg2cffi
+      A C-based driver that requires the PostgreSQL development
+      libraries. Optimal on PyPy and almost indistinguishable from
+      psycopg2 on CPython. Not compatible with gevent.
+
+    pg8000
+     A pure-Python driver suitable for use with gevent. Works on all
+     supported platforms.
+
+     .. note:: pg8000 requires PostgreSQL 9.4 or above for BLOB support.
 
 dsn
     Specifies the data source name for connecting to PostgreSQL.
@@ -54,11 +65,66 @@ The MySQL adapter accepts most parameters supported by the mysqlclient
 library (the maintained version of MySQL-python), including:
 
 driver
-    Either "MySQLdb" (which can be provided by the either of the
-    PyPI distributions `mysqlclient
-    <https://pypi.python.org/pypi/mysqlclient>`_ or `MySQL-python
-    <https://pypi.python.org/pypi/MySQL-python/>`_), or "PyMySQL", or
-    "umysqldb".
+    The possible options are:
+
+    MySQLdb
+      A C-based driver that requires the MySQL client development
+      libraries.. This is best provided by the PyPI distribution
+      `mysqlclient <https://pypi.python.org/pypi/mysqlclient>`_. (It
+      can also be provided by the legacy `MySQL-python
+      <https://pypi.python.org/pypi/MySQL-python/>`_ distribution,
+      but only on CPython 2; this distribution is no longer tested.)
+      These drivers are *not* compatible with gevent.
+
+    PyMySQL
+      A pure-Python driver provided by the distribution of the same
+      name. It works with CPython 2 and 3 and PyPy (where it is
+      preferred). It is compatible with gevent.
+
+    umysqldb
+      A C-based driver that builds on PyMySQL. It is compatible with
+      gevent, but only works on CPython 2. It does not require the
+      MySQL client development libraries but uses a project called
+      ``umysql`` to communicate with the server using only sockets.
+
+      .. note:: Make sure the server has a
+          ``max_allowed_packet`` setting no larger than 16MB. Also
+          make sure that RelStorage's ``blob-chunk-size`` is less than
+          16MB as well.
+
+      .. note:: `This fork of umysqldb
+           <https://github.com/NextThought/umysqldb.git>`_ is
+           recommended. The ``full-buffer`` branch of `this ultramysql
+           fork
+           <https://github.com/NextThought/ultramysql/tree/full-buffer>`_
+           is also recommended if you encounter strange MySQL packet
+           errors.
+
+
+    MySQL Connector/Python
+      This is the `official client
+      <https://dev.mysql.com/doc/connector-python/en/>`_ provided by
+      Oracle. It generally cannot be installed from PyPI or by pip if
+      you want the optional C extension. It has an optional C
+      extension that must be built manually. The C extension (which
+      requires the MySQL client development libraries) performs
+      about as well as mysqlclient, but the pure-python version
+      somewhat slower than PyMySQL. However, it supports more advanced
+      options for failover and high availability.
+
+      When using this name, RelStorage will use the C extension if
+      available, otherwise it will use the Python version.
+
+      Binary packages are distributed by Oracle for many platforms
+      and include the necessary native libraries and C extension.
+
+    C MySQL Connector/Python
+      The same as above, but RelStorage will only use the C extension.
+      This is not compatible with gevent.
+
+    Py MySQL Connector/Python
+      Like the above, but RelStorage will use the pure-Python version
+      only. This is compatible with gevent.
 
 host
     string, host to connect

doc/install.rst

@@ -38,43 +38,43 @@ database.
 
 
 On CPython2, install psycopg2 2.6.1+, mysqlclient 1.3.7+, or cx_Oracle
-5.2+; PyMySQL 0.7 and umysql are also known to work as is pg8000. For
-CPython3, install psycopg2, mysqlclient 1.3.7+ or cx_Oracle; PyMySQL
-and pg8000 are also known to work. On PyPy, install psycopg2cffi
-2.7.4+ or PyMySQL 0.6.6+ (PyPy will generally work with psycopg2 and
-mysqlclient, but it will be *much* slower; in contrast, pg8000
-performs nearly as well. cx_Oracle is untested on PyPy).
-
-Here's a table of known working adapters; adapters **in bold** are the recommended
-adapter; adapters in *italic* are also tested:
-
-========   ================      =================     ======
-Platform   MySQL                 PostgreSQL            Oracle
-========   ================      =================     ======
-CPython2   MySQL-python;         **psycopg2**;         **cx_Oracle**
-           **mysqlclient**;      psycopg2cffi;
-           *PyMySQL*;            *pg8000*
-           umysqldb
-CPython3   **mysqlclient**;      **psycopg2**;         **cx_Oracle**
-           *PyMySQL*             *pg8000*
-PyPy       **PyMySQL**           **psycopg2cffi**;
-                                 *pg8000*
-========   ================      =================     ======
-
-.. note:: If you use umysql, make sure the server has a
-          ``max_allowed_packet`` setting no larger than 16MB. Also
-          make sure that RelStorage's ``blob-chunk-size`` is less than
-          16MB as well.
-
-.. note:: `This fork of umysqldb
-           <https://github.com/NextThought/umysqldb.git>`_ is
-           recommended. The ``full-buffer`` branch of `this ultramysql
-           fork
-           <https://github.com/NextThought/ultramysql/tree/full-buffer>`_
-           is also recommended if you encounter strange MySQL packet
-           errors.
-
-mysqlclient, pg8000 and umysql are compatible (cooperative) with gevent.
+5.2+; PyMySQL 0.7, MySQL Connector/Python 2.1.5 and umysql are also
+known to work as is pg8000.
+
+For CPython3, install psycopg2, mysqlclient or cx_Oracle;
+PyMySQL, MySQL Connector/Python  and pg8000 are also known to work.
+
+On PyPy, install psycopg2cffi 2.7.4+ or PyMySQL 0.6.6+ (PyPy will
+generally work with psycopg2 and mysqlclient, but it will be *much*
+slower; in contrast, pg8000 performs nearly as well. cx_Oracle is
+untested on PyPy).
+
+Here's a table of known (tested) working adapters; adapters **in
+bold** are the recommended adapter.
+
++----------+---------------------+---------------------+--------------+
+| Platform |  MySQL              |   PostgreSQL        |  Oracle      |
++==========+=====================+=====================+==============+
+| CPython2 |                     |  1. **psycopg2**    | **cx_Oracle**|
+|          | 1. **mysqlclient**  |  2. pg8000          |              |
+|          | 2. PyMySQL          |                     |              |
+|          | 3. umysqldb         |                     |              |
+|          | 4. MySQL Connector  |                     |              |
++----------+---------------------+---------------------+--------------+
+| CPython3 | 1. **mysqlclient**  |  1. **psycopg2**    | **cx_Oracle**|
+|          | 2. PyMySQL          |  2. pg8000          |              |
+|          | 3. MySQL Connector  |                     |              |
++----------+---------------------+---------------------+--------------+
+| PyPy     | 1. **PyMySQL**      | 1. **psycopg2cffi** |              |
+|          | 2. MySQL Connector  | 2.  pg8000          |              |
++----------+---------------------+---------------------+--------------+
+
+
+mysqlclient, MySQL Connector/Python (without its C extension), pg8000
+and umysql are compatible (cooperative) with gevent.
+
+For additional details, see the "driver" section for each database in
+:doc:`db-specific-options`.
 
 Memcache Integration
 ====================

relstorage/_compat.py

@@ -13,6 +13,7 @@
 import platform
 
 PY3 = sys.version_info[0] == 3
+PY2 = not PY3
 PYPY = platform.python_implementation() == 'PyPy'
 
 # Dict support
@@ -66,25 +67,17 @@ def list_values(d):
     # buffer on Py3/Py2, respectively, for bytea columns
     _db_binary_types = (memoryview,)
 else:
-    _db_binary_types = (memoryview, buffer)
+    # MySQL Connector/Python returns bytearray, but only from
+    # the Python implementation; the C implementation returns
+    # bytes.
+    _db_binary_types = (memoryview, buffer, bytearray)
 
 def db_binary_to_bytes(data):
     if isinstance(data, _db_binary_types):
         data = bytes(data)
     return data
 
 
-# mysqlclient, a binary driver that works for Py2, Py3 and
-# PyPy (claimed), uses a connection that is a weakref. MySQLdb
-# and PyMySQL use a hard reference
-from weakref import ref as _wref
-def mysql_connection(cursor):
-    conn = cursor.connection
-    if isinstance(conn, _wref):
-        conn = conn()
-    return conn
-
-
 
 from ZODB._compat import BytesIO
 StringIO = BytesIO

relstorage/adapters/_abstract_drivers.py

@@ -68,8 +68,14 @@ def __setattr__(self, name, value):
             return
         return setattr(self.__conn, name, value)
 
-    def cursor(self):
-        return _ConnWrapper(self.__conn.cursor())
+    def cursor(self, *args, **kwargs):
+        return _ConnWrapper(self.__conn.cursor(*args, **kwargs))
+
+    def execute(self, op, args=None):
+        #print(op, args)
+        self.__conn.connection.handle_unread_result()
+        return self.__conn.execute(op, args)
+
 
     def __iter__(self):
         return self.__conn.__iter__()

relstorage/adapters/dbiter.py

@@ -53,7 +53,14 @@ def _transaction_iterator(self, cursor):
         Each row begins with (tid, username, description, extension)
         and may have other columns.
         """
-        for row in cursor:
+        # Iterating the cursor itself in a generator is not safe if
+        # the cursor doesn't actually buffer all the rows *anyway*. If
+        # we break from the iterating loop before exhausting all the
+        # rows, a subsequent query or close operation can lead to
+        # things like MySQL Connector/Python raising
+        # InternalError(unread results)
+        rows = cursor.fetchall()
+        for row in rows:
             tid, username, description, ext = row[:4]
             # Although the transaction interface for username and description are
             # defined as strings, this layer works with bytes. PY3.
@@ -74,7 +81,6 @@ def _transaction_iterator(self, cursor):
 
             yield (tid, username, description, ext) + tuple(row[4:])
 
-
     def iter_transactions(self, cursor):
         """Iterate over the transaction log, newest first.
 

relstorage/adapters/mover.py

@@ -77,9 +77,13 @@ def load_current(self, cursor, oid):
         stmt = self._load_current_query
 
         cursor.execute(stmt, (oid,))
-        if cursor.rowcount:
-            assert cursor.rowcount == 1
-            state, tid = cursor.fetchone()
+        # Note that we cannot rely on cursor.rowcount being
+        # a valid indicator. The DB-API doesn't require it, and
+        # some implementations, like MySQL Connector/Python are
+        # unbuffered by default and can't provide it.
+        row = cursor.fetchone()
+        if row:
+            state, tid = row
             state = db_binary_to_bytes(state)
             # If it's None, the object's creation has been
             # undone.
@@ -102,9 +106,9 @@ def load_revision(self, cursor, oid, tid):
         """
         stmt = self._load_revision_query
         cursor.execute(stmt, (oid, tid))
-        if cursor.rowcount:
-            assert cursor.rowcount == 1
-            (state,) = cursor.fetchone()
+        row = cursor.fetchone()
+        if row:
+            (state,) = row
             return db_binary_to_bytes(state)
         return None
 
@@ -120,7 +124,8 @@ def exists(self, cursor, oid):
         """Returns a true value if the given object exists."""
         stmt = self._exists_query
         cursor.execute(stmt, (oid,))
-        return cursor.rowcount
+        row = cursor.fetchone()
+        return row
 
     @metricmethod_sampled
     def load_before(self, cursor, oid, tid):
@@ -137,9 +142,9 @@ def load_before(self, cursor, oid, tid):
         LIMIT 1
         """
         cursor.execute(stmt, (oid, tid))
-        if cursor.rowcount:
-            assert cursor.rowcount == 1
-            state, tid = cursor.fetchone()
+        row = cursor.fetchone()
+        if row:
+            state, tid = row
             state = db_binary_to_bytes(state)
             # None in state means The object's creation has been undone
             return state, tid
@@ -162,9 +167,9 @@ def get_object_tid_after(self, cursor, oid, tid):
         LIMIT 1
         """
         cursor.execute(stmt, (oid, tid))
-        if cursor.rowcount:
-            assert cursor.rowcount == 1
-            return cursor.fetchone()[0]
+        row = cursor.fetchone()
+        if row:
+            return row[0]
         else:
             return None