Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix psycopg2 issues with database creation #494

Closed
wants to merge 2 commits into from
Closed

Fix psycopg2 issues with database creation #494

wants to merge 2 commits into from

Conversation

glumia
Copy link

@glumia glumia commented Jan 22, 2021 •
edited

This closes #494.

@glumia glumia marked this pull request as draft January 23, 2021 07:28
@glumia
Copy link
Author

glumia commented Jan 23, 2021 •
edited

Test tests/functions/test_database.py::TestDatabasePostgresPg8000::test_create_and_drop is failing because of issues with sqlalchemy and pg8000, check sqlalchemy/sqlalchemy@b2f2bf1.

We should either set 1.16.5 as max pg8000 version or find some way to make versions >=1.16.6 work with sqlalchemy 1.3.x.

I have no idea about how the test was passing before my changes ahah
Was it passing at all? Or was the CI broken because of it?

Update
I just checked, yes the CI is broken because of that.

Update 2
More info about this here: sqlalchemy/sqlalchemy#5645

As suggested in the linked sqlalchemy we could (in the meanwhile that either sqlalchemy 1.4.x is released or that an upper bound is set to pg8000's version on sqlalchemy 1.3.x) override pg8000's dialect description_encoding value.

@glumia glumia mentioned this pull request Jan 23, 2021
Closed
@glumia glumia changed the title Fix psycopg2 issues with database creation Fix CI (pg8000) and psycopg2 issues with database creation Jan 23, 2021
@glumia glumia marked this pull request as ready for review January 23, 2021 11:18
@glumia glumia force-pushed the master branch 2 times, most recently from 3f43e05 to 4d418aa Compare January 23, 2021 15:59
@glumia glumia changed the title Fix CI (pg8000) and psycopg2 issues with database creation Fix psycopg2 issues with database creation Feb 4, 2021
@glumia
Copy link
Author

glumia commented Feb 4, 2021

Issue #495 that broke CI was solved by release 1.3.23 of sqlalchemy, I removed the workaround 4d418aa.

Simplify code
af4f413 
pyscopg2 supports the isolation_level parameter 'AUTOCOMMIT' at engine
creation, there's no need to configure manually the connection.
nsoranzo added a commit to nsoranzo/sqlalchemy-utils that referenced this pull request Mar 18, 2021
@nsoranzo
Use with engine.connect() context manager
4f52578 
Consolidate autocommit across drivers that support it, xref.
kvesteri#494 and
fix kvesteri#486 .
@nsoranzo nsoranzo mentioned this pull request Mar 22, 2021
Merged
kvesteri added a commit that referenced this pull request Apr 12, 2021
@nsoranzo @bernt-matthias
@cnu @lucventurini @BradenM @kvesteri
SQLAlchemy 1.4 support + Move CI to GitHub workflows (#506)
a4154bd 
* fix database existence check

follow up to #372

* always dispose engine after db existence check

* add docs for postgres_db parameter

* fix dialect_name

Co-authored-by: Nicola Soranzo <nicola.soranzo@gmail.com>

* optimize execution order

Co-authored-by: Nicola Soranzo <nicola.soranzo@gmail.com>

* database_exists fix return

- postgres: return for the first positive test
- use immutable for default argument

* use None as default

Co-authored-by: Nicola Soranzo <nicola.soranzo@gmail.com>

* break if successful

Co-authored-by: Nicola Soranzo <nicola.soranzo@gmail.com>

* dispose only for exception

* use Null connection pool and close connections

- use a connection (which is closed automatically) for
  data base existence check
- explicitely use Null connection pool

already with the 1st change disposal of the engine (which closes all
open connections) is not necessary anymore. with the second change
we are completely sure.

* fix isort call in tox.ini and import order

* rename parameter to databases

* move functions to module level

* Add support for SQLAlchemy 1.4

Import _ColumnEntity from sqlalchemy.orm.context if importing from
.query fails.

And while checking if database_exists, use url.set() as URL object is
now immutable.

* Fixed errors related to URL.database not being directly settable.

* Removed the try..except constructs to set the database. Now checking *once* the version of sqlalchemy in use and deciding *once* how to change the database address. Using a wrapping function ('set_database_from_url') to simplify the code.

* Cleanups

* Use `execution_options()` method of `Connection`

to set the transaction isolation level to autocommit for PostgreSQL.

* Lint fixes

* Move CI lint job to GitHub workflows

* Testing: drop Python 3.5 and Python 3.9

* Use `with engine.connect()` context manager

Consolidate autocommit across drivers that support it, xref.
#494 and
fix #486 .

* Move CI tests from TravisCI to GitHub workflows

* Add SQLAlchemy 1.3 on Python 3.6 to the test matrix

* Update tests for changes in `create_database`

to use `with engine.connect()` context manager (commit
4f52578 )

* Install the version of pg8000 recommended by SQLAlchemy 1.3

when testing.

xref. #500

Fix the following exception:

```
_____________________________________________________________________________ TestDatabasePostgresPg8000.test_create_and_drop _____________________________________________________________________________

self = <tests.functions.test_database.TestDatabasePostgresPg8000 object at 0x7f3b3da76490>, dsn = 'postgresql+pg8000://postgres:postgres@localhost/db_to_test_create_and_drop_via_pg8000_driver'

    def test_create_and_drop(self, dsn):
>       assert not database_exists(dsn)

tests/functions/test_database.py:15:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
sqlalchemy_utils/functions/database.py:488: in database_exists
    return bool(_get_scalar_result(engine, text))
sqlalchemy_utils/functions/database.py:443: in _get_scalar_result
    with engine.connect() as conn:
.venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py:2263: in connect
    return self._connection_cls(self, **kwargs)
.venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py:104: in __init__
    else engine.raw_connection()
.venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py:2369: in raw_connection
    return self._wrap_pool_connect(
.venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py:2336: in _wrap_pool_connect
    return fn()
.venv/lib/python3.8/site-packages/sqlalchemy/pool/base.py:304: in unique_connection
    return _ConnectionFairy._checkout(self)
.venv/lib/python3.8/site-packages/sqlalchemy/pool/base.py:778: in _checkout
    fairy = _ConnectionRecord.checkout(pool)
.venv/lib/python3.8/site-packages/sqlalchemy/pool/base.py:495: in checkout
    rec = pool._do_get()
.venv/lib/python3.8/site-packages/sqlalchemy/pool/impl.py:241: in _do_get
    return self._create_connection()
.venv/lib/python3.8/site-packages/sqlalchemy/pool/base.py:309: in _create_connection
    return _ConnectionRecord(self)
.venv/lib/python3.8/site-packages/sqlalchemy/pool/base.py:440: in __init__
    self.__connect(first_connect_check=True)
.venv/lib/python3.8/site-packages/sqlalchemy/pool/base.py:664: in __connect
    pool.dispatch.first_connect.for_modify(
.venv/lib/python3.8/site-packages/sqlalchemy/event/attr.py:314: in exec_once_unless_exception
    self._exec_once_impl(True, *args, **kw)
.venv/lib/python3.8/site-packages/sqlalchemy/event/attr.py:285: in _exec_once_impl
    self(*args, **kw)
.venv/lib/python3.8/site-packages/sqlalchemy/event/attr.py:322: in __call__
    fn(*args, **kw)
.venv/lib/python3.8/site-packages/sqlalchemy/util/langhelpers.py:1406: in go
    return once_fn(*arg, **kw)
.venv/lib/python3.8/site-packages/sqlalchemy/engine/strategies.py:199: in first_connect
    dialect.initialize(c)
.venv/lib/python3.8/site-packages/sqlalchemy/dialects/postgresql/pg8000.py:215: in initialize
    super(PGDialect_pg8000, self).initialize(connection)
.venv/lib/python3.8/site-packages/sqlalchemy/dialects/postgresql/base.py:2624: in initialize
    super(PGDialect, self).initialize(connection)
.venv/lib/python3.8/site-packages/sqlalchemy/engine/default.py:311: in initialize
    self.server_version_info = self._get_server_version_info(
.venv/lib/python3.8/site-packages/sqlalchemy/dialects/postgresql/base.py:2869: in _get_server_version_info
    v = connection.execute("select version()").scalar()
.venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py:1003: in execute
    return self._execute_text(object_, multiparams, params)
.venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py:1172: in _execute_text
    ret = self._execute_context(
.venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py:1316: in _execute_context
    self._handle_dbapi_exception(
.venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py:1514: in _handle_dbapi_exception
    util.raise_(exc_info[1], with_traceback=exc_info[2])
.venv/lib/python3.8/site-packages/sqlalchemy/util/compat.py:182: in raise_
    raise exception
.venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py:1294: in _execute_context
    result = context._setup_crud_result_proxy()
.venv/lib/python3.8/site-packages/sqlalchemy/engine/default.py:1258: in _setup_crud_result_proxy
    result = self.get_result_proxy()
.venv/lib/python3.8/site-packages/sqlalchemy/engine/default.py:1233: in get_result_proxy
    return result.ResultProxy(self)
.venv/lib/python3.8/site-packages/sqlalchemy/engine/result.py:775: in __init__
    self._init_metadata()
.venv/lib/python3.8/site-packages/sqlalchemy/engine/result.py:807: in _init_metadata
    self._metadata = ResultMetaData(self, cursor_description)
.venv/lib/python3.8/site-packages/sqlalchemy/engine/result.py:290: in __init__
    raw = self._merge_cursor_description(
.venv/lib/python3.8/site-packages/sqlalchemy/engine/result.py:496: in _merge_cursor_description
    return [
.venv/lib/python3.8/site-packages/sqlalchemy/engine/result.py:496: in <listcomp>
    return [
.venv/lib/python3.8/site-packages/sqlalchemy/engine/result.py:616: in _merge_cols_by_none
    for (
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <sqlalchemy.engine.result.ResultMetaData object at 0x7f3b3daa4a90>, context = <sqlalchemy.dialects.postgresql.pg8000.PGExecutionContext_pg8000 object at 0x7f3b3dda7100>
cursor_description = [('version', 25, None, None, None, None, ...)]

    def _colnames_from_description(self, context, cursor_description):
        """Extract column names and data types from a cursor.description.

        Applies unicode decoding, column translation, "normalization",
        and case sensitivity rules to the names based on the dialect.

        """

        dialect = context.dialect
        case_sensitive = dialect.case_sensitive
        translate_colname = context._translate_colname
        description_decoder = (
            dialect._description_decoder
            if dialect.description_encoding
            else None
        )
        normalize_name = (
            dialect.normalize_name if dialect.requires_name_normalize else None
        )
        untranslated = None

        self.keys = []

        for idx, rec in enumerate(cursor_description):
            colname = rec[0]
            coltype = rec[1]

            if description_decoder:
>               colname = description_decoder(colname)
E               TypeError: expected bytes, str found

.venv/lib/python3.8/site-packages/sqlalchemy/engine/result.py:545: TypeError
```

* Fix another import for SQLAlchemy 1.4

* Fix test failing on SQLAlchemy 1.4

* Fix `AttributeError: 'Query' object has no attribute '_entities'`

* Fix `AttributeError: type object 'User' has no attribute '_decl_class_registry'`

and similar messages for other objects.

* Extend linting to all Python files

* Fix `AttributeError: 'Query' object has no attribute '_mapper_zero'`

* Fix `AttributeError: module 'sqlalchemy.orm.mapper' has no attribute '_mapper_registry'`

* Fix `make_order_by_deterministic` for SQLAlchemy 1.4

Fix errors:

```
    def make_order_by_deterministic(query):
        """

        order_by_func = sa.asc

>       if not query._order_by:
E       AttributeError: 'Query' object has no attribute '_order_by'

```

```
tests/functions/test_make_order_by_deterministic.py:74:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

query = <sqlalchemy.orm.query.Query object at 0x7f03d5af64f0>

    def make_order_by_deterministic(query):

                else:
                    order_by_func = sa.asc
>               column = order_by.get_children()[0]
E               TypeError: 'itertools.chain' object is not subscriptable
```

* Add message to assert

* Debug test failures under SQLAlchemy 1.4

* Don't mask potential `AttributeError` exceptions in `sa.engine.URL.create()`

* Fix wrong `dialect_name` check

Co-authored-by: Nicola Soranzo <nicola.soranzo@gmail.com>

* Remove `u` unnecessary on Python 3

* Drop unnecessary `alias()` call

This returns a `Subquery` object which is not executable in SQLAlchemy 1.4:

```
tests/relationships/test_select_correlated_expression.py:399:
    assert session.execute(aggregate)
.venv/lib/python3.8/site-packages/sqlalchemy/orm/session.py:1587: in execute
    statement = coercions.expect(roles.StatementRole, statement)
.venv/lib/python3.8/site-packages/sqlalchemy/sql/coercions.py:200: in expect
    return impl._implicit_coercions(
.venv/lib/python3.8/site-packages/sqlalchemy/sql/coercions.py:836: in _implicit_coercions
    return super(StatementImpl, self)._implicit_coercions(
.venv/lib/python3.8/site-packages/sqlalchemy/sql/coercions.py:242: in _implicit_coercions
    self._raise_for_expected(element, argname, resolved)
.venv/lib/python3.8/site-packages/sqlalchemy/sql/coercions.py:270: in _raise_for_expected
    util.raise_(exc.ArgumentError(msg, code=code), replace_context=err)

E           sqlalchemy.exc.ArgumentError: Executable SQL or text() construct expected, got <sqlalchemy.sql.selectable.Subquery at 0x7fb34b5e1880; test>.
```

* fix get_columns tests

* fix instant defaults listener

* remove sort_query and get_query_entities

The sort_query and get_query_entities functions never worked fully as intended and
contained lots of quirky  edge cases:

1. sort_query function was dangerous in a sense that it could be used for really
inefficient queries (sorting by non-indexed column).
2. The entity string label introspection in both functions relied on SQLAlchemy internals
which were drastically changed in SA 1.4. Relying on those was never a good idea in
the first place.

* Remove also `get_query_entity_by_alias()`

which depends on `get_query_entities()` which was removed in commit
460e1da .

* Fix linting errors

* fix scalar_subquery warnings

* Update sqlalchemy_utils/compat.py

Co-authored-by: Nicola Soranzo <nicola.soranzo@gmail.com>

* fix selectable columns warning

* Add health checks for SQL Server docker service

Remove `!` from password so `SA_PASSWORD` doesn't need extra quotes.

* Fix install_mssql.sh commands

* Use `isolation_level` argument also for `mssql`

* Restore `engine.dispose()`

* Add `coverage.xml` to `.gitignore`

* change tests to use non-deprecated class

Change EncryptedType tests to use StringEncryptedType

* use scalar subquery for sqlalchemy 1.4

* use create_mock_engine for SA 1.4

* isort linting fixes

Co-authored-by: Matthias Bernt <m.bernt@ufz.de>
Co-authored-by: Srinivasan R <srinivasanr@gmail.com>
Co-authored-by: Luca Venturini <luca.venturini@earlham.ac.uk>
Co-authored-by: Braden Mars <bradenmars@bradenmars.me>
Co-authored-by: Konsta Vesterinen <konsta.vesterinen@gmail.com>
@glumia
Copy link
Author

glumia commented Oct 25, 2021

Closing this one as the issue was fixed by a4154bd.

@glumia glumia closed this Oct 25, 2021
StdCarrot pushed a commit to StdCarrot/sqlalchemy-utils that referenced this pull request Apr 14, 2022
@nsoranzo @bernt-matthias
@cnu @lucventurini @BradenM @kvesteri
SQLAlchemy 1.4 support + Move CI to GitHub workflows (kvesteri#506)
13aae8e 
* fix database existence check

follow up to kvesteri#372

* always dispose engine after db existence check

* add docs for postgres_db parameter

* fix dialect_name

Co-authored-by: Nicola Soranzo <nicola.soranzo@gmail.com>

* optimize execution order

Co-authored-by: Nicola Soranzo <nicola.soranzo@gmail.com>

* database_exists fix return

- postgres: return for the first positive test
- use immutable for default argument

* use None as default

Co-authored-by: Nicola Soranzo <nicola.soranzo@gmail.com>

* break if successful

Co-authored-by: Nicola Soranzo <nicola.soranzo@gmail.com>

* dispose only for exception

* use Null connection pool and close connections

- use a connection (which is closed automatically) for
  data base existence check
- explicitely use Null connection pool

already with the 1st change disposal of the engine (which closes all
open connections) is not necessary anymore. with the second change
we are completely sure.

* fix isort call in tox.ini and import order

* rename parameter to databases

* move functions to module level

* Add support for SQLAlchemy 1.4

Import _ColumnEntity from sqlalchemy.orm.context if importing from
.query fails.

And while checking if database_exists, use url.set() as URL object is
now immutable.

* Fixed errors related to URL.database not being directly settable.

* Removed the try..except constructs to set the database. Now checking *once* the version of sqlalchemy in use and deciding *once* how to change the database address. Using a wrapping function ('set_database_from_url') to simplify the code.

* Cleanups

* Use `execution_options()` method of `Connection`

to set the transaction isolation level to autocommit for PostgreSQL.

* Lint fixes

* Move CI lint job to GitHub workflows

* Testing: drop Python 3.5 and Python 3.9

* Use `with engine.connect()` context manager

Consolidate autocommit across drivers that support it, xref.
kvesteri#494 and
fix kvesteri#486 .

* Move CI tests from TravisCI to GitHub workflows

* Add SQLAlchemy 1.3 on Python 3.6 to the test matrix

* Update tests for changes in `create_database`

to use `with engine.connect()` context manager (commit
4f52578 )

* Install the version of pg8000 recommended by SQLAlchemy 1.3

when testing.

xref. kvesteri#500

Fix the following exception:

```
_____________________________________________________________________________ TestDatabasePostgresPg8000.test_create_and_drop _____________________________________________________________________________

self = <tests.functions.test_database.TestDatabasePostgresPg8000 object at 0x7f3b3da76490>, dsn = 'postgresql+pg8000://postgres:postgres@localhost/db_to_test_create_and_drop_via_pg8000_driver'

    def test_create_and_drop(self, dsn):
>       assert not database_exists(dsn)

tests/functions/test_database.py:15:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
sqlalchemy_utils/functions/database.py:488: in database_exists
    return bool(_get_scalar_result(engine, text))
sqlalchemy_utils/functions/database.py:443: in _get_scalar_result
    with engine.connect() as conn:
.venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py:2263: in connect
    return self._connection_cls(self, **kwargs)
.venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py:104: in __init__
    else engine.raw_connection()
.venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py:2369: in raw_connection
    return self._wrap_pool_connect(
.venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py:2336: in _wrap_pool_connect
    return fn()
.venv/lib/python3.8/site-packages/sqlalchemy/pool/base.py:304: in unique_connection
    return _ConnectionFairy._checkout(self)
.venv/lib/python3.8/site-packages/sqlalchemy/pool/base.py:778: in _checkout
    fairy = _ConnectionRecord.checkout(pool)
.venv/lib/python3.8/site-packages/sqlalchemy/pool/base.py:495: in checkout
    rec = pool._do_get()
.venv/lib/python3.8/site-packages/sqlalchemy/pool/impl.py:241: in _do_get
    return self._create_connection()
.venv/lib/python3.8/site-packages/sqlalchemy/pool/base.py:309: in _create_connection
    return _ConnectionRecord(self)
.venv/lib/python3.8/site-packages/sqlalchemy/pool/base.py:440: in __init__
    self.__connect(first_connect_check=True)
.venv/lib/python3.8/site-packages/sqlalchemy/pool/base.py:664: in __connect
    pool.dispatch.first_connect.for_modify(
.venv/lib/python3.8/site-packages/sqlalchemy/event/attr.py:314: in exec_once_unless_exception
    self._exec_once_impl(True, *args, **kw)
.venv/lib/python3.8/site-packages/sqlalchemy/event/attr.py:285: in _exec_once_impl
    self(*args, **kw)
.venv/lib/python3.8/site-packages/sqlalchemy/event/attr.py:322: in __call__
    fn(*args, **kw)
.venv/lib/python3.8/site-packages/sqlalchemy/util/langhelpers.py:1406: in go
    return once_fn(*arg, **kw)
.venv/lib/python3.8/site-packages/sqlalchemy/engine/strategies.py:199: in first_connect
    dialect.initialize(c)
.venv/lib/python3.8/site-packages/sqlalchemy/dialects/postgresql/pg8000.py:215: in initialize
    super(PGDialect_pg8000, self).initialize(connection)
.venv/lib/python3.8/site-packages/sqlalchemy/dialects/postgresql/base.py:2624: in initialize
    super(PGDialect, self).initialize(connection)
.venv/lib/python3.8/site-packages/sqlalchemy/engine/default.py:311: in initialize
    self.server_version_info = self._get_server_version_info(
.venv/lib/python3.8/site-packages/sqlalchemy/dialects/postgresql/base.py:2869: in _get_server_version_info
    v = connection.execute("select version()").scalar()
.venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py:1003: in execute
    return self._execute_text(object_, multiparams, params)
.venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py:1172: in _execute_text
    ret = self._execute_context(
.venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py:1316: in _execute_context
    self._handle_dbapi_exception(
.venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py:1514: in _handle_dbapi_exception
    util.raise_(exc_info[1], with_traceback=exc_info[2])
.venv/lib/python3.8/site-packages/sqlalchemy/util/compat.py:182: in raise_
    raise exception
.venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py:1294: in _execute_context
    result = context._setup_crud_result_proxy()
.venv/lib/python3.8/site-packages/sqlalchemy/engine/default.py:1258: in _setup_crud_result_proxy
    result = self.get_result_proxy()
.venv/lib/python3.8/site-packages/sqlalchemy/engine/default.py:1233: in get_result_proxy
    return result.ResultProxy(self)
.venv/lib/python3.8/site-packages/sqlalchemy/engine/result.py:775: in __init__
    self._init_metadata()
.venv/lib/python3.8/site-packages/sqlalchemy/engine/result.py:807: in _init_metadata
    self._metadata = ResultMetaData(self, cursor_description)
.venv/lib/python3.8/site-packages/sqlalchemy/engine/result.py:290: in __init__
    raw = self._merge_cursor_description(
.venv/lib/python3.8/site-packages/sqlalchemy/engine/result.py:496: in _merge_cursor_description
    return [
.venv/lib/python3.8/site-packages/sqlalchemy/engine/result.py:496: in <listcomp>
    return [
.venv/lib/python3.8/site-packages/sqlalchemy/engine/result.py:616: in _merge_cols_by_none
    for (
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <sqlalchemy.engine.result.ResultMetaData object at 0x7f3b3daa4a90>, context = <sqlalchemy.dialects.postgresql.pg8000.PGExecutionContext_pg8000 object at 0x7f3b3dda7100>
cursor_description = [('version', 25, None, None, None, None, ...)]

    def _colnames_from_description(self, context, cursor_description):
        """Extract column names and data types from a cursor.description.

        Applies unicode decoding, column translation, "normalization",
        and case sensitivity rules to the names based on the dialect.

        """

        dialect = context.dialect
        case_sensitive = dialect.case_sensitive
        translate_colname = context._translate_colname
        description_decoder = (
            dialect._description_decoder
            if dialect.description_encoding
            else None
        )
        normalize_name = (
            dialect.normalize_name if dialect.requires_name_normalize else None
        )
        untranslated = None

        self.keys = []

        for idx, rec in enumerate(cursor_description):
            colname = rec[0]
            coltype = rec[1]

            if description_decoder:
>               colname = description_decoder(colname)
E               TypeError: expected bytes, str found

.venv/lib/python3.8/site-packages/sqlalchemy/engine/result.py:545: TypeError
```

* Fix another import for SQLAlchemy 1.4

* Fix test failing on SQLAlchemy 1.4

* Fix `AttributeError: 'Query' object has no attribute '_entities'`

* Fix `AttributeError: type object 'User' has no attribute '_decl_class_registry'`

and similar messages for other objects.

* Extend linting to all Python files

* Fix `AttributeError: 'Query' object has no attribute '_mapper_zero'`

* Fix `AttributeError: module 'sqlalchemy.orm.mapper' has no attribute '_mapper_registry'`

* Fix `make_order_by_deterministic` for SQLAlchemy 1.4

Fix errors:

```
    def make_order_by_deterministic(query):
        """

        order_by_func = sa.asc

>       if not query._order_by:
E       AttributeError: 'Query' object has no attribute '_order_by'

```

```
tests/functions/test_make_order_by_deterministic.py:74:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

query = <sqlalchemy.orm.query.Query object at 0x7f03d5af64f0>

    def make_order_by_deterministic(query):

                else:
                    order_by_func = sa.asc
>               column = order_by.get_children()[0]
E               TypeError: 'itertools.chain' object is not subscriptable
```

* Add message to assert

* Debug test failures under SQLAlchemy 1.4

* Don't mask potential `AttributeError` exceptions in `sa.engine.URL.create()`

* Fix wrong `dialect_name` check

Co-authored-by: Nicola Soranzo <nicola.soranzo@gmail.com>

* Remove `u` unnecessary on Python 3

* Drop unnecessary `alias()` call

This returns a `Subquery` object which is not executable in SQLAlchemy 1.4:

```
tests/relationships/test_select_correlated_expression.py:399:
    assert session.execute(aggregate)
.venv/lib/python3.8/site-packages/sqlalchemy/orm/session.py:1587: in execute
    statement = coercions.expect(roles.StatementRole, statement)
.venv/lib/python3.8/site-packages/sqlalchemy/sql/coercions.py:200: in expect
    return impl._implicit_coercions(
.venv/lib/python3.8/site-packages/sqlalchemy/sql/coercions.py:836: in _implicit_coercions
    return super(StatementImpl, self)._implicit_coercions(
.venv/lib/python3.8/site-packages/sqlalchemy/sql/coercions.py:242: in _implicit_coercions
    self._raise_for_expected(element, argname, resolved)
.venv/lib/python3.8/site-packages/sqlalchemy/sql/coercions.py:270: in _raise_for_expected
    util.raise_(exc.ArgumentError(msg, code=code), replace_context=err)

E           sqlalchemy.exc.ArgumentError: Executable SQL or text() construct expected, got <sqlalchemy.sql.selectable.Subquery at 0x7fb34b5e1880; test>.
```

* fix get_columns tests

* fix instant defaults listener

* remove sort_query and get_query_entities

The sort_query and get_query_entities functions never worked fully as intended and
contained lots of quirky  edge cases:

1. sort_query function was dangerous in a sense that it could be used for really
inefficient queries (sorting by non-indexed column).
2. The entity string label introspection in both functions relied on SQLAlchemy internals
which were drastically changed in SA 1.4. Relying on those was never a good idea in
the first place.

* Remove also `get_query_entity_by_alias()`

which depends on `get_query_entities()` which was removed in commit
460e1da .

* Fix linting errors

* fix scalar_subquery warnings

* Update sqlalchemy_utils/compat.py

Co-authored-by: Nicola Soranzo <nicola.soranzo@gmail.com>

* fix selectable columns warning

* Add health checks for SQL Server docker service

Remove `!` from password so `SA_PASSWORD` doesn't need extra quotes.

* Fix install_mssql.sh commands

* Use `isolation_level` argument also for `mssql`

* Restore `engine.dispose()`

* Add `coverage.xml` to `.gitignore`

* change tests to use non-deprecated class

Change EncryptedType tests to use StringEncryptedType

* use scalar subquery for sqlalchemy 1.4

* use create_mock_engine for SA 1.4

* isort linting fixes

Co-authored-by: Matthias Bernt <m.bernt@ufz.de>
Co-authored-by: Srinivasan R <srinivasanr@gmail.com>
Co-authored-by: Luca Venturini <luca.venturini@earlham.ac.uk>
Co-authored-by: Braden Mars <bradenmars@bradenmars.me>
Co-authored-by: Konsta Vesterinen <konsta.vesterinen@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
1 participant
@glumia