Create RedshiftDialectMixin class. Add Psycopg2CFFIRedshiftDialect by Brooke-white · Pull Request #231 · sqlalchemy-redshift/sqlalchemy-redshift

Brooke-white · 2021-08-26T20:35:13Z

Builds upon graingert’s work in #100 to refactor dialect code to utilize a RedshiftDialectMixin class to define driver specific dialects.

Additionally, the PyscopgRedshiftDialectMixin class is defined and used as a base class for psycopg flavored dialects. It holds the implementation for the create_connect_args() method, which previously lived in theRedshiftDialect class. My thoughts here are that this class could also be utilized in supporting a psycopg3 dialect in the future.

While graingert’s work in #100 added support for additional dialects (pg8000, pypostgresql, zxjdbc), I believe supporting these drivers would require some additional effort. Support for these drivers is not included in this PR.

Tests are parameterized using pytest.fixture to run on each defined dialect (i.e. redshift, redshift+psycopg2, redshift+psycopg2cffi). Unit tests utilize a parameterized ”stub” redshift dialect pytest.fixture to avoid needing an Amazon Redshift cluster. Unsure if we want to run all tests using both redshift and redshift+psycopg2, as they are using the same dialect — so I’ll leave this up to the reviewers :)

All tests (including ones requiring a Redshift cluster) have been run with nominal results.

Todos

MIT compatible
Tests
Documentation
Updated CHANGES.rst

jklukas

From a first read-through, this looks pretty comprehensive. I'd like to see some documentation about the existence of the CFFI variant, and I left some comments about naming. I wish I were better acquainted with modern sqlalchemy conventions to make stronger suggestions on naming.

Perhaps @zzzeek would be willing to do a quick drive-by about how the class naming and registration looks here.

jklukas · 2021-08-30T19:50:00Z

sqlalchemy_redshift/__init__.py


-registry.register("redshift", "sqlalchemy_redshift.dialect", "RedshiftDialect")
 registry.register(
-    "redshift.psycopg2", "sqlalchemy_redshift.dialect", "RedshiftDialect"


Do we have any idea whether removing RedshiftDialect could affect existing user code? It's not clear to me where this name would get referenced.

For users directly using RedshiftDialect class, this would be a breaking change. Searching GitHub, I found a number of projects reference this class directly who would impacted. see here. So we probably want to take a different approach here.

One option to avoid a breaking change here could be to re-add RedshiftDialect having it inherit from Pyscopg2RedshiftDialect. Or we simply re-name Pyscopg2RedshiftDialect -> RedshiftDialect.

I do like modernizing the names here, so I think it's good to add some shim for compatibility rather than staying stuck with the old name.

I think we can get away with something like this in dialect.py:

# Add RedshiftDialect synonym for backwards compatibility. RedshiftDialect = RedshiftDialect_psycopg2

Addressed in ae43dfa

jklukas · 2021-08-30T19:51:15Z

sqlalchemy_redshift/dialect.py

        return all_constraints


+class PsycopgRedshiftDialectMixin(RedshiftDialectMixin):


Suggested change

class PsycopgRedshiftDialectMixin(RedshiftDialectMixin):

class Psycopg2RedshiftDialectMixin(RedshiftDialectMixin):

I assume that the missing "2" here is a typo, but let me know if there's additional nuance here.

good catch -- will fix this

addressed in 96b9af0

jklukas · 2021-08-30T19:52:45Z

sqlalchemy_redshift/dialect.py


+class PsycopgRedshiftDialectMixin(RedshiftDialectMixin):
+    """
+    Define Psycopg specific behavior.


Suggested change

Define Psycopg specific behavior.

Define behavior specific to ``psycopg2``.

addressed in 17c6a6b

jklukas · 2021-08-30T19:54:38Z

sqlalchemy_redshift/dialect.py

-from sqlalchemy.dialects.postgresql.psycopg2 import PGDialect_psycopg2
+from sqlalchemy.dialects.postgresql import (
+    psycopg2, psycopg2cffi


Are PGDialect_psycopg2 and psycopg2 synonyms? Do we know why the longer name exists?

Are PGDialect_psycopg2 and psycopg2 synonyms?

From Sqlalchemy source PGDialect_psycopg2,sqlalchemy.dialects.postrgresql.psycopg2.dialect, and sqlalchemy-dialects.postgres.__init__.py, I believe so.

Do we know why the longer name exists?

The name still exists. There is an equivelent name for PGDialect_psycopg2cffi. I can modify this import to be the following if it's preferred:

from sqlalchemy.dialects.postgresql.psycopg2 import PGDialect_psycopg2 from sqlalchemy.dialects.postgresql.psycopg2cffi import PGDialect_psycopg2cffi

I see now that you inherit from psycopg2.dialect later on, and indeed that's a synonym of PGDialect_psycopg2.

I would prefer, though, that we use the class names. It feels more natural for expressing class inheritance. And there will be some more obvious symmetry, too, with our own class names assuming we update to use RedshiftDialect_psycopg2, etc.

addressed in 291d88e

jklukas · 2021-08-30T20:00:54Z

sqlalchemy_redshift/dialect.py

+        return cargs, default_args
+
+
+class Psycopg2RedshiftDialect(


I'd like to following existing naming conventions in SQLAlchemy where possible. From the postgresql dialect code, it looks like this would be called RedshiftDialect_psycopg2. Does that seem like a reasonable choice here?

Yes, sounds good. I will make this change

addressed in 29d5704

zzzeek · 2021-08-30T21:34:56Z

I havent looked at everything but what stands out to me is the test suite refactoring. not sure how feasible this is but assuming redshift is using SQLAlchemy's test harness, we dont generally have dialect names hardcoded in our test suite as this would not scale, and the commandline runner can automatically run the tests against any number of dialects. instead of targeting "dialect = XDialect()" everywhere you instead look at the global "sqlalchemy.testing.config.db.dialect" for the current dialect. Directives associated with test classes like __backend__ = True will automatically run class-based test suites for every "dialect" currently selected.

our commandline to run the tests for postgresql looks like:

tox -e py3-postgresql

which will then run a pytest like this:

python -m pytest --db postgresql --dbdriver psycopg2 --dbdriver asyncpg --dbdriver pg8000

the suite will generate URLs given each driver and run the test suite against all the above DBAPIs.

Just something to think about as sqlalchemy-redshift is definitely going to want to support other DBAPIs, at least one of asyncpg or psycopg3 for async support, for example.

…shiftDialectMixin

…2, Psycopg2CFFIRedshiftDialect -> RedshiftDialect_psycopg2cffi

…ility

…g class inheritance

Brooke-white

thank you both for the feedback!

As for documentation regarding the CFFI variant, from what I saw in the sqlalchemy source, PGDialect_psycopg2cffi seems very similar to PGDialect_psycopg2, save for some methods having logic around extras/extensions and the package's versioning.

It looks like @zzzeek reviewed a PR associated with PGDialect_psycopg2cffi, sqlalchemy #3052. I see some mentions of differences in unicode bind parameter names and floating point values, but I am far from knowledgable on this subject and this PR is from quite a few years ago, so things have probably changed.

Regarding the test suite, I don't believe sqlalchemy-redshift uses the sqlalchemy test harness at this point in time, but after taking a quick look I agree--it seems like a much more scalable (and clean) approach going forward. This could be worth looking into more in the future.

Brooke-white · 2021-08-30T22:45:35Z

sqlalchemy_redshift/dialect.py

-from sqlalchemy.dialects.postgresql.psycopg2 import PGDialect_psycopg2
+from sqlalchemy.dialects.postgresql import (
+    psycopg2, psycopg2cffi


addressed in 291d88e

Brooke-white · 2021-08-30T22:46:19Z

sqlalchemy_redshift/dialect.py

        return all_constraints


+class PsycopgRedshiftDialectMixin(RedshiftDialectMixin):


addressed in 96b9af0

Brooke-white · 2021-08-30T22:46:30Z

sqlalchemy_redshift/dialect.py


+class PsycopgRedshiftDialectMixin(RedshiftDialectMixin):
+    """
+    Define Psycopg specific behavior.


addressed in 17c6a6b

Brooke-white · 2021-08-30T22:46:44Z

sqlalchemy_redshift/dialect.py

+        return cargs, default_args
+
+
+class Psycopg2RedshiftDialect(


addressed in 29d5704

jklukas · 2021-09-01T20:51:58Z

Thanks for all the updates, @Brooke-white. I have it on my to-do list to give this a full review, but probably won't happen until Friday.

jklukas · 2021-09-07T15:14:50Z

Thanks for all the updates, @Brooke-white. I have it on my to-do list to give this a full review, but probably won't happen until Friday.

Even more things came up last week, so apologies that this will need to be further delayed.

Brooke-white · 2021-09-08T16:42:57Z

Thanks for the update @jklukas , do you have an ETA for when you'll be able to give a full review?

jklukas

Changes look good, including the new names and the shims for compatibility with RedshiftDialect. I've kicked off integration tests and will merge assuming those all come back clean.

Thomas Grainger and others added 14 commits August 26, 2021 12:43

Add support for all SQLAlchemy pg drivers Fixes sqlalchemy-redshift#8

2133b22

fix tests

65b0e74

fix(dialect, RedshiftDialectMixin): remove references to RedshiftDialect

695eccf

fix(tests): accomidate RedshiftDialectMixin

da35101

tests(dialect): fix failing tests

8443626

refactor(dialect): remove untested dialects

46ad8f5

tests(dialect): remove dialects

12e6fd0

refactor(test, dependencies): remove untested dialects

3ce5055

refactor: lint, code cleanup

34917b3

test(dialect): custom dialect class inheritance

097375f

test(dialect_types): fix bug

ae9cb8d

chore: add entry to changelog

2190108

chore: update changelog

407deb0

chore: apply lint

0268621

jklukas requested review from graingert and jklukas August 26, 2021 20:48

jklukas reviewed Aug 30, 2021

View reviewed changes

Brooke-white added 7 commits August 30, 2021 15:41

refactor(dialect, PsycopgRedshiftDialectMixin): Rename to Psycopg2Red…

96b9af0

…shiftDialectMixin

docs(Psycopg2RedshiftDialectMixin): reword doc string

17c6a6b

refactor(dialect): Pyscopg2RedshiftDialect -> RedshiftDialect_psycopg…

29d5704

…2, Psycopg2CFFIRedshiftDialect -> RedshiftDialect_psycopg2cffi

refactor(dialect): add RedshiftDialect synonym for backwards compatib…

ae43dfa

…ility

refactor(dialect, inheritance): Use PG_Dialect_ classes for expressin…

291d88e

…g class inheritance

chore: lint

d76bd96

chore: update changelog wording

49aa3ce

Brooke-white commented Aug 30, 2021

View reviewed changes

Brooke-white requested a review from jklukas September 1, 2021 20:10

jklukas approved these changes Sep 9, 2021

View reviewed changes

jklukas merged commit 7cbb98b into sqlalchemy-redshift:main Sep 9, 2021

		return all_constraints


		class PsycopgRedshiftDialectMixin(RedshiftDialectMixin):

	class PsycopgRedshiftDialectMixin(RedshiftDialectMixin):
	class Psycopg2RedshiftDialectMixin(RedshiftDialectMixin):

	Define Psycopg specific behavior.
	Define behavior specific to ``psycopg2``.

Conversation

Brooke-white commented Aug 26, 2021

Todos

Uh oh!

jklukas left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Brooke-white Aug 30, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zzzeek commented Aug 30, 2021

Uh oh!

Brooke-white left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jklukas commented Sep 1, 2021

Uh oh!

jklukas commented Sep 7, 2021

Uh oh!

Brooke-white commented Sep 8, 2021

Uh oh!

jklukas left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Brooke-white Aug 30, 2021 •

edited

Loading

Brooke-white left a comment •

edited

Loading