Support inspection of Redshift datatypes

[Brooke-white]

This PR modifies RedshiftDialectMixin to support database schema inspection of Amazon Redshift datatypes. It adds entries in ischema_names for these Redshift specific datatypes. It also makes a small change to the definition of Timetz and Timestamptz datatype definition for compatibility with SqlAlchemy insepctor.

Use case: Support for Redshift datatypes in Querybook Metastore

Issue: When inspect.get_columns(...) is called, NullType() is returned as the type for columns with type geometry and super, which causes issues in downstream tools like Querybook's metastore.

Repro:

from sqlalchemy.engine import reflection
from sqlalchemy import create_engine
engine = create_engine('redshift+redshift_connector://...')
inspect = reflection.Inspector.from_engine(engine)
x = inspect.get_columns(table_name='my_table', schema='public')

where my_table is defined as

create table public.my_table (c1 geometry, c2 super, c3 timetz, c4 timestamptz);

For columns with type timetz and timestamptz, sa.dialects.postgresql.TIME and sa.dialects.postgresql.TIMESTAMP are returned, which isn't ideal since we've defined TIMETZ and TIMESTAMPTZ datatypes within sqlalchemy-redshift.

sqlalchemy.engine.dialects.postgresql.base.PGDialect._get_column_info uses the ischema_names to map datatype string names to datatype classes.

screenshots below show the behavior before & after this change when the above use case is run:

note entries 11, 12, 15, 16

behavior before:

behavior after:

Todos

• [ x ] MIT compatible
• [ x ] Tests
• [ x ] Documentation
• [ x ] Updated CHANGES.rst

[jklukas]

Left some early thoughts. Feel free to force push the content to the trigger-integration branch of the main repo when you're ready to have integration tests run.

[jklukas]

Can we define this constant in UPPER_CAMEL_CASE and move it to the top of the file with other constants?

[Brooke-white]

I added a few more changes here as I noticed there were still some issues with type inspection

• Adds RedshiftTypeEngine so we can support calling str on custom data types (e.g. SUPER, TIMETZ) (which requires compilation of the type ref).
• Overrides sqlalchemy.sql.type_api.TypeEngine._default_dialect() method to return RedshiftDialectMixin rather than DefaultDialect — as we need access to RedshiftTypeCompiler to successfully compile the type. See sqlalchemy.sql.type_api.TypeEngine.compile

Use case:

from sqlalchemy_redshift.dialect import TIMETZ, SUPER, TIMESTAMPTZ, GEOMETRY

for x in (TIMETZ(), TIMESTAMPTZ(), SUPER(), GEOMETRY()):
    print(x)

TIMETZ
TIMESTAMPTZ
SUPER
GEOMETRY

previous behavior

sqlalchemy.exc.UnsupportedCompilationError: Compiler <sqlalchemy.sql.compiler.GenericTypeCompiler object at 0x7fdd6427e610> can't render element of type <class 'sqlalchemy_redshift.dialect.TIMETZ'> (Background on this error at: http://sqlalche.me/e/l7de)

• Has RedshiftDialectMixin inherit from DefaultDialect so the default dialect initializer is called upon creation of a RedshiftDialectMixin object, so dialect attributes such as type_compiler are initialized. (the use case is the above mentioned type compilation)

• Modifies Redshift data types to inherit from RedshiftTypeEngine so compilation of these types results in the RedshiftDialectMixin being utilized.

• Adds docs for TIMETZ and TIMESTAMPTZ’s unused timezone parameter

• Changes redshift_ischema_names -> REDSHIFT_ISCHEMA_NAMES

I don’t think we can move REDSHIFT_ISCHEMA_NAMES to be defined with the rest of the constants at the top of dialect.py as it makes reference to the Redshift data types (e.g. TIMETZ) which are defined after the constants. I’m open to any suggestions as to where else this constant could live

[Brooke-white]

Feel free to force push the content to the trigger-integration branch of the main repo when you're ready to have integration tests run.

Hmm, I pushed to trigger-integration but encountered some message saying build have been temporarily disabled. I'll pick this PR back up following Thanksgiving week. Enjoy the holiday! :)

[jklukas]

but encountered some message saying build have been temporarily disabled

I'll contact Travis support to get this unstuck.

Enjoy the holiday! :)

Likewise!

[Brooke-white]

Thanks for getting Travis going again, @jklukas! Integration tests are all passing. Am I ok to merge?

[jklukas]

Looks good! Feel free to merge.