Engine introspection is not working with SQLAlchemy==1.3.24

[bkyryliuk]

Repro:

from sqlalchemy import create_engine
from sqlalchemy.engine.reflection import Inspector
engine = create_engine(f'databricks://token:{TOCKEN}@dropbox-prod.cloud.databricks.com:443/default?catalog=hive_metastore', connect_args={"http_path":"/sql/1.0/warehouses/{ID}"})
inspector = Inspector.from_engine(engine)
columns = inspector.get_columns(f"{catalog}.{schema}.{table}")

Error mgs:

AttributeError: 'DatabricksDialect' object has no attribute 'get_driver_connection'

[susodapop]

Thanks for this report. I've opened a pull request that should fix this for you. You can even test it out using a dev build I just published to pypi: pip install databricks-sql-connector==2.7.1.dev1

You need to modify your repro to use the connection string that databricks-sql-python expects (this is addressed in the PR description), but otherwise it runs like it should.

from sqlalchemy import create_engine
from sqlalchemy.engine.reflection import Inspector


host="****.cloud.databricks.com"
http_path="/sql/1.0/warehouses/****"
access_token="dapi****"
catalog="samples"
schema="nyctaxi"

engine = create_engine(f'databricks://token:{access_token}@{host}', connect_args={"http_path": http_path, "catalog": catalog, "schema": schema})
inspector = Inspector.from_engine(engine)
columns = inspector.get_columns("trips", schema="nyctaxi")

Note that you should include the schema in your calls to Inspector.get_columns(). I even added a smoke test to our e2e suite so we can tell if this breaks in the future.