Port to psycopg3 (#1324)

* WIP. * Add some comments about porting from psycopg 2 to 3 (#1318) * WIP * Disable _set_wait_callback() * TransactionStatus. * First working query. * More pg3 changes. * test_pgexecute still fails. * Fix bytea support. * Fix json and enum unicode. * Get unit tests to pass. * Behave tests still break, WIP. * Prompt seems to be displayed fine, why don't the tests see the whitespace? * Python version. * Fix test. * Black. * Added black to dev reqs. * nbu link for donations. * Use psycopg.sql to format statement. * Special case for show help in pgbouncer. * Fix test. * Added integration test. * Install pgbouncer in ci. * Fix integration test. * Remove tmate session. * Revert commenting out python versions. * Pin pgspecial to >=2. * Changelog. Co-authored-by: Daniele Varrazzo <daniele.varrazzo@gmail.com> Co-authored-by: Amjith Ramanujam <amjith.r@gmail.com>
dbcli · Jun 6, 2022 · 1807175 · 1807175
1 parent 372da81
commit 1807175
Show file tree

Hide file tree

Showing 21 changed files with 273 additions and 350 deletions.
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
@@ -11,7 +11,7 @@ jobs:
 
     strategy:
       matrix:
-        python-version: [3.6, 3.7, 3.8, 3.9]
+        python-version: ["3.7", "3.8", "3.9", "3.10"]
 
     services:
       postgres:
@@ -35,6 +35,35 @@ jobs:
         with:
           python-version: ${{ matrix.python-version }}
 
+      - name: Install pgbouncer
+        run: |
+            sudo apt install pgbouncer -y
+
+            sudo chmod 666 /etc/pgbouncer/*.*
+
+            cat <<EOF > /etc/pgbouncer/userlist.txt
+            "postgres" "postgres"
+            EOF
+
+            cat <<EOF > /etc/pgbouncer/pgbouncer.ini
+            [databases]
+            * = host=localhost port=5432
+            [pgbouncer]
+            listen_port = 6432
+            listen_addr = localhost
+            auth_type = trust
+            auth_file = /etc/pgbouncer/userlist.txt
+            logfile = pgbouncer.log
+            pidfile = pgbouncer.pid
+            admin_users = postgres
+            EOF
+
+            sudo systemctl stop pgbouncer
+
+            pgbouncer -d /etc/pgbouncer/pgbouncer.ini
+
+            psql -h localhost -U postgres -p 6432 pgbouncer -c 'show help'
+
       - name: Install requirements
         run: |
           pip install -U pip setuptools
@@ -56,8 +85,8 @@ jobs:
         run: rst2html.py --halt=warning changelog.rst >/dev/null
 
       - name: Run Black
-        run: pip install black && black --check .
-        if: matrix.python-version == '3.6'
+        run: black --check .
+        if: matrix.python-version == '3.7'
 
       - name: Coverage
         run: |

diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
@@ -1,6 +1,5 @@
 repos:
 -   repo: https://github.com/psf/black
-    rev: 21.5b0
+    rev: 22.3.0
     hooks:
     - id: black
-
diff --git a/README.rst b/README.rst
@@ -11,6 +11,7 @@ Picture by @fomenko_ph (Telegram).
 
 Please consider donating or volunteering.
 
+* https://bank.gov.ua/en/
 * https://savelife.in.ua/en/donate/
 * https://www.comebackalive.in.ua/donate
 * https://www.globalgiving.org/projects/ukraine-crisis-relief-fund/
@@ -51,10 +52,7 @@ If you already know how to install python packages, then you can simply do:
 If you don't know how to install python packages, please check the
 `detailed instructions`_.
 
-If you are restricted to using psycopg2 2.7.x then pip will try to install it from a binary. There are some known issues with the psycopg2 2.7 binary - see the `psycopg docs`_ for more information about this and how to force installation from source. psycopg2 2.8 has fixed these problems, and will build from source.
-
 .. _`detailed instructions`: https://github.com/dbcli/pgcli#detailed-installation-instructions
-.. _`psycopg docs`: http://initd.org/psycopg/docs/install.html#change-in-binary-packages-between-psycopg-2-7-and-2-8
 
 Usage
 -----
@@ -353,8 +351,8 @@ choice:
 
     In [3]: my_result = _
 
-Pgcli only runs on Python3.6+ since 2.2.0, if you use an old version of Python,
-you should use install ``pgcli <= 2.2.0``.
+Pgcli only runs on Python3.7+ since 4.0.0, if you use an old version of Python,
+you should use install ``pgcli <= 4.0.0``.
 
 Thanks:
 -------
@@ -368,7 +366,7 @@ of this app.
 `Click <http://click.pocoo.org/>`_ is used for command line option parsing
 and printing error messages.
 
-Thanks to `psycopg <http://initd.org/psycopg/>`_ for providing a rock solid
+Thanks to `psycopg <https://www.psycopg.org/>`_ for providing a rock solid
 interface to Postgres database.
 
 Thanks to all the beta testers and contributors for your time and patience. :)

diff --git a/changelog.rst b/changelog.rst
@@ -1,3 +1,11 @@
+Upcoming:
+=========
+
+Internal:
+---------
+
+* Port to psycopg3 (https://github.com/psycopg/psycopg). Needs a major version bump.
+
 3.4.1 (2022/03/19)
 ==================
 

diff --git a/pgcli/main.py b/pgcli/main.py
@@ -1,12 +1,7 @@
-import platform
-import warnings
-
 from configobj import ConfigObj, ParseError
 from pgspecial.namedqueries import NamedQueries
 from .config import skip_initial_comment
 
-warnings.filterwarnings("ignore", category=UserWarning, module="psycopg2")
-
 import atexit
 import os
 import re
@@ -22,7 +17,6 @@
 import platform
 from time import time, sleep
 from typing import Optional
-from urllib.parse import urlparse
 
 keyring = None  # keyring will be loaded later
 
@@ -80,11 +74,9 @@
     from urllib.parse import urlparse, unquote, parse_qs
 
 from getpass import getuser
-from psycopg2 import OperationalError, InterfaceError
 
-# pg3: https://www.psycopg.org/psycopg3/docs/api/conninfo.html
-from psycopg2.extensions import make_dsn, parse_dsn
-import psycopg2
+from psycopg import OperationalError, InterfaceError
+from psycopg.conninfo import make_conninfo, conninfo_to_dict
 
 from collections import namedtuple
 
@@ -537,7 +529,7 @@ def connect_service(self, service, user):
         )
 
     def connect_uri(self, uri):
-        kwargs = psycopg2.extensions.parse_dsn(uri)
+        kwargs = conninfo_to_dict(uri)
         remap = {"dbname": "database", "password": "passwd"}
         kwargs = {remap.get(k, k): v for k, v in kwargs.items()}
         self.connect(**kwargs)
@@ -585,7 +577,7 @@ def connect(
         if not passwd and keyring:
 
             try:
-                passwd = keyring.get_password("pgcli", key)
+                passwd = keyring.get_password("pgcli", key) or ""
             except (RuntimeError, keyring.errors.InitError) as e:
                 click.secho(
                     keyring_error_message.format(
@@ -608,7 +600,7 @@ def should_ask_for_password(exc):
             return False
 
         if dsn:
-            parsed_dsn = parse_dsn(dsn)
+            parsed_dsn = conninfo_to_dict(dsn)
             if "host" in parsed_dsn:
                 host = parsed_dsn["host"]
             if "port" in parsed_dsn:
@@ -655,7 +647,7 @@ def should_ask_for_password(exc):
             port = self.ssh_tunnel.local_bind_ports[0]
 
             if dsn:
-                dsn = make_dsn(dsn, host=host, port=port)
+                dsn = make_conninfo(dsn, host=host, port=port)
 
         # Attempt to connect to the database.
         # Note that passwd may be empty on the first attempt. If connection
@@ -1208,7 +1200,7 @@ def echo_via_pager(self, text, color=None):
 
 
 @click.command()
-# Default host is '' so psycopg2 can default to either localhost or unix socket
+# Default host is '' so psycopg can default to either localhost or unix socket
 @click.option(
     "-h",
     "--host",
@@ -1606,18 +1598,11 @@ def format_status(cur, status):
         if hasattr(cur, "description"):
             column_types = []
             for d in cur.description:
-                # pg3: type_name = cur.adapters.types[d.type_code].name
-                if (
-                    # pg3: type_name in ("numeric", "float4", "float8")
-                    d[1] in psycopg2.extensions.DECIMAL.values
-                    or d[1] in psycopg2.extensions.FLOAT.values
-                ):
+                col_type = cur.adapters.types.get(d.type_code)
+                type_name = col_type.name if col_type else None
+                if type_name in ("numeric", "float4", "float8"):
                     column_types.append(float)
-                if (
-                    # pg3: type_name in ("int2", "int4", "int8")
-                    d[1] == psycopg2.extensions.INTEGER.values
-                    or d[1] in psycopg2.extensions.LONGINTEGER.values
-                ):
+                if type_name in ("int2", "int4", "int8"):
                     column_types.append(int)
                 else:
                     column_types.append(str)
@@ -1634,7 +1619,11 @@ def format_status(cur, status):
             and headers
         ):
             formatted = formatter.format_output(
-                cur, headers, format_name="vertical", column_types=None, **output_kwargs
+                cur,
+                headers,
+                format_name="vertical",
+                column_types=column_types,
+                **output_kwargs,
             )
             if isinstance(formatted, str):
                 formatted = iter(formatted.splitlines())