Port from psycopg2 to psycopg3 - #88
Merged
Merged
Conversation
psycopg2 is in maintenance mode; psycopg3 is where development happens, has per-connection adaptation, automatic prepared statements, and an async story if we ever want one. The evaluation (35-step parity harness, benchmarks against a scratch server and devmirror) found behavior byte-identical and performance at parity after the tuning below. Library: - psycopg2.sql -> psycopg.sql everywhere (API-compatible) - execute_values -> executemany with the VALUES template expanded in _execute (pipelined in psycopg3); no caller changes - cursor.copy_from/copy_to/copy_expert -> cursor.copy() streaming, via a new _copy_from_stdin helper (base, table, statstable call sites) - the psycopg2 adaptation protocol (register_adapter/ISQLQuote/adapt, Json subclass, register_json, new_type) -> psycopg3 Dumper/Loader classes registered per connection in setup_connection; custom types are dumped as text with unknown oid so the server infers types from context, matching psycopg2's client-side interpolation semantics - mogrify only exists on ClientCursor: PostgresBase._mogrify creates one on demand for logging and error messages - DDL cannot take bound parameters under server-side binding: sequence creation in _copy_from and index storage parameters now inline literals - typed parameter arrays can change plans or lack operators: $in/$nin cast ANY(%s) to the column's type, and _create_typecast covers $overlaps and bare array equality (psycopg3 picks smallint[] for a list of small ints; integer[] = smallint[] has no operator) - ServerCursor.itersize set to 2000 to match psycopg2's named cursors - join_search's qualify() used the psycopg2-only Composed.seq and Identifier.string attributes - extras pgsource/pgbinary now install psycopg / psycopg[binary] Tests and CI: - test imports moved to psycopg.sql / psycopg errors - Array wrapper now renders postgres array literals rather than ARRAY[...] expressions (server-side parameters are values, not SQL); the adapter-protocol test is replaced by literal-rendering cases - Json wrapper contract test replaces the psycopg2-subclass test - composable introspection in test_utils uses iteration and as_string() (Composed.seq / SQL.string / Identifier.strings do not exist in psycopg3) - the packaging job's import guard now checks the psycopg[binary] hint Verified: 519 passed / 24 skipped under psycopg 3.3.4 on PostgreSQL 18; byte-identical 35-step workload parity against the base branch running psycopg2; all 21 devmirror tests pass against the live mirror. The downstream.yml jobs stay red until lmfdb and seminars port their own psycopg2.sql imports (paired PRs, as with the extras transition). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This was referenced Jul 20, 2026
Author
|
The driver-tolerant downstream PRs are open — each works with psycodict on either side of this transition (they probe
Once lmfdb's and seminars' PRs merge to their mains, the |
- analyze() still called cursor.mogrify, which only ClientCursor has under psycopg3, so every call raised AttributeError before running EXPLAIN; it now uses the _mogrify helper. (Only the printed query needed interpolating -- EXPLAIN itself accepts bound parameters under server-side binding, which the new tests confirm.) - copy_to() rejected the null= option that psycopg2's copy_to (and psycodict's passthrough of it) supported; the COPY statement now carries NULL alongside DELIMITER, mirroring copy_from and reload. Regression tests for both: analyze prints the interpolated query and a plan (with and without execution), and a copy_to/copy_from round trip with null='NULL' -- including a value that merely starts with the marker. analyze also re-verified against devmirror, where the report reproduced the failure. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
AndrewVSutherland
pushed a commit
to AndrewVSutherland/psetpartners
that referenced
this pull request
Jul 20, 2026
Two changes preparing for psycodict's 1.0 release: - The psycodict submodule moves from 35f09b4 (2021-era) to current master (9e65584), which includes the removal of the search/extras table split, the new test suite, and a long run of bug fixes. psetpartners is unaffected by the extras removal: it never used extra_columns/extrafile, and its projection=3 calls survive (3 is kept as an alias for all-columns-plus-id). All psycodict API it uses -- PostgresDatabase(config), DelayCommit, IdentifierWrapper, Configuration, search/lucky/lookup/update -- is unchanged. - dbwrapper.py imports SQL from psycodict rather than psycopg2.sql. psycodict is switching from psycopg2 to psycopg3 (roed314/psycodict#88), and composed fragments must come from the driver psycodict is built on; psycodict re-exports SQL from the right place on both sides of that transition, so this one-line change makes the code driver-agnostic. When the submodule is later bumped past the port, the psycopg package must be installed in the deployment environment in place of psycopg2 (neither is listed in requirements.txt; the driver is installed out of band). Verified by importing through the submodule path with the pinned commit (SQL resolves to psycopg2.sql, composition works) and with the psycopg3 branch checked out in the submodule (SQL resolves to psycopg.sql). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This was referenced Jul 20, 2026
sage.all.RealNumber is the create_RealNumber factory, not a class, so registering the dumper on it raised TypeError under Sage; import the class from sage.rings.real_mpfr instead (as encoding.py already does). The lmfdb-env action stripped LMFDB's psycodict requirement line -- which, now that it reads psycodict[pgbinary], was the only source of the psycopg driver -- and then installed psycodict with --no-deps; install with the pgbinary extra instead. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
roed314
added a commit
to LMFDB/lmfdb
that referenced
this pull request
Jul 21, 2026
psycodict is switching from psycopg2 to psycopg3 (roed314/psycodict#88). Query fragments composed in the LMFDB are executed by psycodict's _execute, so they must come from whichever driver psycodict is built on -- and since both drivers can be installed at once, try/except imports are not a valid probe. The new lmfdb/utils/psycopg_compat.py keys off psycodict's re-exported SQL and provides SQL, Identifier, Placeholder, Literal, Composable, Composed plus DatabaseError, DataError, NumericValueOutOfRange and QueryCanceledError from the matching driver. All sixteen direct psycopg2 imports (search/verify/knowledge/users/api/groups/ecnf, the dynamic-knowls test and the old belyi script) now import from it, so the LMFDB works unchanged on either side of psycodict's transition; once psycodict is pinned past it, the module body can shrink to plain psycopg imports. requirements.txt now asks for psycodict[pgbinary], which installs whichever driver the checked-out psycodict needs (psycopg2-binary today, psycopg[binary] after the port) instead of naming psycopg2 directly. Verified by loading the compat module against both psycodict versions: all names resolve from the matching driver, composed statements execute through _execute, and a statement_timeout raises the exported QueryCanceledError under both -- that exception is what turns timeouts into the 'search took too long' page rather than a 500. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
roed314
added a commit
to roed314/seminars
that referenced
this pull request
Jul 21, 2026
psycodict is switching from psycopg2 to psycopg3 (roed314/psycodict#88). Composed query fragments are executed by psycodict's _execute, so they must come from whichever driver psycodict is built on; both drivers can be installed at once, so the new seminars/psycopg_compat.py keys off psycodict's re-exported SQL and provides SQL, Composed, Identifier, Literal, Placeholder and DatabaseError from the matching driver. All direct psycopg2 imports (utils, seminar, talk, create/main, lock, importing/sanitize) now go through it, so the site works on either side of the transition; once psycodict is pinned past it, the module body shrinks to plain psycopg imports. sanitize.py's raw cursor.copy_to (an API removed in psycopg3) becomes copy_table_to_file in the compat module, with a branch per driver. Replacing it also fixes a latent bug: the manual column quoting it used has been broken since psycopg2 2.9, which quotes column names itself -- the export raised UndefinedColumn ('column ""id"" does not exist') on any modern psycopg2. requirements.txt now asks for psycodict[pgbinary], which installs whichever driver the checked-out psycodict needs, instead of naming psycopg2-binary directly. Verified by running the compat module against both psycodict versions over a live server: names resolve from the matching driver, composed statements execute through _execute, and copy_table_to_file produces byte-identical exports (including delimiter escaping) under both. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements the driver switch evaluated for #27, on top of the extras removal (#59). psycopg2 is in maintenance mode; psycopg3 is where development happens, gives per-connection adaptation (no more process-global
register_adapter), automatic server-side prepared statements, and an async option on the same codebase if ever wanted.What changes
psycopg2.sql→psycopg.sqleverywhere (API-compatible), and psycodict now re-exportsSQL,Identifier,Placeholder,Literal,Composable,Composedso downstream projects can compose queries without importing a driver directly.execute_values→executemanywith theVALUEStemplate expanded inside_execute(pipelined in psycopg3); no caller changes.cursor.copy_from/copy_to/copy_expert→ thecursor.copy()streaming API, via a new_copy_from_stdinhelper.register_adapter/ISQLQuote/adapt, theJsonsubclass,register_json,new_type) → psycopg3 Dumper/Loader classes registered per connection. Custom types are dumped as text with unknown oid, so the server infers types from context exactly as it did for psycopg2's client-side literals — this is why behavior is unchanged.mogrifyonly exists onClientCursor: aPostgresBase._mogrifyhelper creates one on demand for logging/error messages._copy_fromand index storage parameters now inline literals.smallint[]for a list of small ints, so$in/$nincastANY(%s)to the column's type (also a large plan-quality win, per the evaluation), and_create_typecastnow covers$overlapsand bare array equality (integer[] = smallint[]has no operator).ServerCursor.itersizeset to 2000 to match psycopg2's named cursors (default 100 multiplies round trips).join_search'squalify()used psycopg2-onlyComposed.seq/Identifier.string.pgsource/pgbinarynow installpsycopg/psycopg[binary]— names unchanged, so lmfdb-lite and the CI need no edits.as_string(); theArraywrapper now renders postgres array literals ({1,2,3}) rather thanARRAY[...]expressions — under server-side binding a parameter is a value, not SQL; the packaging job's import-guard checks the newpsycopg[binary]hint.Verification
Merge ordering
The
downstream.ymljobs will stay red on this PR until the paired downstream PRs merge: lmfdb, seminars, and psetpartners each importpsycopg2.sqldirectly today. Those PRs are driver-tolerant (they follow whichever driver the installed psycodict uses), so the safe order is: merge the downstream PRs first (they work fine against current master), at which point this PR's downstream jobs go green, then merge this. Links will be added below as they open.Caveat for the release notes: the Sage-mode dumpers (Sage Integer / RealLiteral) are written but exercised only via the Sage CI job — worth a look during review.
🤖 Generated with Claude Code