Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
SQLAlchemy 1.4 support + Move CI to GitHub workflows (#506)
* fix database existence check follow up to #372 * always dispose engine after db existence check * add docs for postgres_db parameter * fix dialect_name Co-authored-by: Nicola Soranzo <nicola.soranzo@gmail.com> * optimize execution order Co-authored-by: Nicola Soranzo <nicola.soranzo@gmail.com> * database_exists fix return - postgres: return for the first positive test - use immutable for default argument * use None as default Co-authored-by: Nicola Soranzo <nicola.soranzo@gmail.com> * break if successful Co-authored-by: Nicola Soranzo <nicola.soranzo@gmail.com> * dispose only for exception * use Null connection pool and close connections - use a connection (which is closed automatically) for data base existence check - explicitely use Null connection pool already with the 1st change disposal of the engine (which closes all open connections) is not necessary anymore. with the second change we are completely sure. * fix isort call in tox.ini and import order * rename parameter to databases * move functions to module level * Add support for SQLAlchemy 1.4 Import _ColumnEntity from sqlalchemy.orm.context if importing from .query fails. And while checking if database_exists, use url.set() as URL object is now immutable. * Fixed errors related to URL.database not being directly settable. * Removed the try..except constructs to set the database. Now checking *once* the version of sqlalchemy in use and deciding *once* how to change the database address. Using a wrapping function ('set_database_from_url') to simplify the code. * Cleanups * Use `execution_options()` method of `Connection` to set the transaction isolation level to autocommit for PostgreSQL. * Lint fixes * Move CI lint job to GitHub workflows * Testing: drop Python 3.5 and Python 3.9 * Use `with engine.connect()` context manager Consolidate autocommit across drivers that support it, xref. #494 and fix #486 . * Move CI tests from TravisCI to GitHub workflows * Add SQLAlchemy 1.3 on Python 3.6 to the test matrix * Update tests for changes in `create_database` to use `with engine.connect()` context manager (commit 4f52578 ) * Install the version of pg8000 recommended by SQLAlchemy 1.3 when testing. xref. #500 Fix the following exception: ``` _____________________________________________________________________________ TestDatabasePostgresPg8000.test_create_and_drop _____________________________________________________________________________ self = <tests.functions.test_database.TestDatabasePostgresPg8000 object at 0x7f3b3da76490>, dsn = 'postgresql+pg8000://postgres:postgres@localhost/db_to_test_create_and_drop_via_pg8000_driver' def test_create_and_drop(self, dsn): > assert not database_exists(dsn) tests/functions/test_database.py:15: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sqlalchemy_utils/functions/database.py:488: in database_exists return bool(_get_scalar_result(engine, text)) sqlalchemy_utils/functions/database.py:443: in _get_scalar_result with engine.connect() as conn: .venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py:2263: in connect return self._connection_cls(self, **kwargs) .venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py:104: in __init__ else engine.raw_connection() .venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py:2369: in raw_connection return self._wrap_pool_connect( .venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py:2336: in _wrap_pool_connect return fn() .venv/lib/python3.8/site-packages/sqlalchemy/pool/base.py:304: in unique_connection return _ConnectionFairy._checkout(self) .venv/lib/python3.8/site-packages/sqlalchemy/pool/base.py:778: in _checkout fairy = _ConnectionRecord.checkout(pool) .venv/lib/python3.8/site-packages/sqlalchemy/pool/base.py:495: in checkout rec = pool._do_get() .venv/lib/python3.8/site-packages/sqlalchemy/pool/impl.py:241: in _do_get return self._create_connection() .venv/lib/python3.8/site-packages/sqlalchemy/pool/base.py:309: in _create_connection return _ConnectionRecord(self) .venv/lib/python3.8/site-packages/sqlalchemy/pool/base.py:440: in __init__ self.__connect(first_connect_check=True) .venv/lib/python3.8/site-packages/sqlalchemy/pool/base.py:664: in __connect pool.dispatch.first_connect.for_modify( .venv/lib/python3.8/site-packages/sqlalchemy/event/attr.py:314: in exec_once_unless_exception self._exec_once_impl(True, *args, **kw) .venv/lib/python3.8/site-packages/sqlalchemy/event/attr.py:285: in _exec_once_impl self(*args, **kw) .venv/lib/python3.8/site-packages/sqlalchemy/event/attr.py:322: in __call__ fn(*args, **kw) .venv/lib/python3.8/site-packages/sqlalchemy/util/langhelpers.py:1406: in go return once_fn(*arg, **kw) .venv/lib/python3.8/site-packages/sqlalchemy/engine/strategies.py:199: in first_connect dialect.initialize(c) .venv/lib/python3.8/site-packages/sqlalchemy/dialects/postgresql/pg8000.py:215: in initialize super(PGDialect_pg8000, self).initialize(connection) .venv/lib/python3.8/site-packages/sqlalchemy/dialects/postgresql/base.py:2624: in initialize super(PGDialect, self).initialize(connection) .venv/lib/python3.8/site-packages/sqlalchemy/engine/default.py:311: in initialize self.server_version_info = self._get_server_version_info( .venv/lib/python3.8/site-packages/sqlalchemy/dialects/postgresql/base.py:2869: in _get_server_version_info v = connection.execute("select version()").scalar() .venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py:1003: in execute return self._execute_text(object_, multiparams, params) .venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py:1172: in _execute_text ret = self._execute_context( .venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py:1316: in _execute_context self._handle_dbapi_exception( .venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py:1514: in _handle_dbapi_exception util.raise_(exc_info[1], with_traceback=exc_info[2]) .venv/lib/python3.8/site-packages/sqlalchemy/util/compat.py:182: in raise_ raise exception .venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py:1294: in _execute_context result = context._setup_crud_result_proxy() .venv/lib/python3.8/site-packages/sqlalchemy/engine/default.py:1258: in _setup_crud_result_proxy result = self.get_result_proxy() .venv/lib/python3.8/site-packages/sqlalchemy/engine/default.py:1233: in get_result_proxy return result.ResultProxy(self) .venv/lib/python3.8/site-packages/sqlalchemy/engine/result.py:775: in __init__ self._init_metadata() .venv/lib/python3.8/site-packages/sqlalchemy/engine/result.py:807: in _init_metadata self._metadata = ResultMetaData(self, cursor_description) .venv/lib/python3.8/site-packages/sqlalchemy/engine/result.py:290: in __init__ raw = self._merge_cursor_description( .venv/lib/python3.8/site-packages/sqlalchemy/engine/result.py:496: in _merge_cursor_description return [ .venv/lib/python3.8/site-packages/sqlalchemy/engine/result.py:496: in <listcomp> return [ .venv/lib/python3.8/site-packages/sqlalchemy/engine/result.py:616: in _merge_cols_by_none for ( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <sqlalchemy.engine.result.ResultMetaData object at 0x7f3b3daa4a90>, context = <sqlalchemy.dialects.postgresql.pg8000.PGExecutionContext_pg8000 object at 0x7f3b3dda7100> cursor_description = [('version', 25, None, None, None, None, ...)] def _colnames_from_description(self, context, cursor_description): """Extract column names and data types from a cursor.description. Applies unicode decoding, column translation, "normalization", and case sensitivity rules to the names based on the dialect. """ dialect = context.dialect case_sensitive = dialect.case_sensitive translate_colname = context._translate_colname description_decoder = ( dialect._description_decoder if dialect.description_encoding else None ) normalize_name = ( dialect.normalize_name if dialect.requires_name_normalize else None ) untranslated = None self.keys = [] for idx, rec in enumerate(cursor_description): colname = rec[0] coltype = rec[1] if description_decoder: > colname = description_decoder(colname) E TypeError: expected bytes, str found .venv/lib/python3.8/site-packages/sqlalchemy/engine/result.py:545: TypeError ``` * Fix another import for SQLAlchemy 1.4 * Fix test failing on SQLAlchemy 1.4 * Fix `AttributeError: 'Query' object has no attribute '_entities'` * Fix `AttributeError: type object 'User' has no attribute '_decl_class_registry'` and similar messages for other objects. * Extend linting to all Python files * Fix `AttributeError: 'Query' object has no attribute '_mapper_zero'` * Fix `AttributeError: module 'sqlalchemy.orm.mapper' has no attribute '_mapper_registry'` * Fix `make_order_by_deterministic` for SQLAlchemy 1.4 Fix errors: ``` def make_order_by_deterministic(query): """ order_by_func = sa.asc > if not query._order_by: E AttributeError: 'Query' object has no attribute '_order_by' ``` ``` tests/functions/test_make_order_by_deterministic.py:74: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ query = <sqlalchemy.orm.query.Query object at 0x7f03d5af64f0> def make_order_by_deterministic(query): else: order_by_func = sa.asc > column = order_by.get_children()[0] E TypeError: 'itertools.chain' object is not subscriptable ``` * Add message to assert * Debug test failures under SQLAlchemy 1.4 * Don't mask potential `AttributeError` exceptions in `sa.engine.URL.create()` * Fix wrong `dialect_name` check Co-authored-by: Nicola Soranzo <nicola.soranzo@gmail.com> * Remove `u` unnecessary on Python 3 * Drop unnecessary `alias()` call This returns a `Subquery` object which is not executable in SQLAlchemy 1.4: ``` tests/relationships/test_select_correlated_expression.py:399: assert session.execute(aggregate) .venv/lib/python3.8/site-packages/sqlalchemy/orm/session.py:1587: in execute statement = coercions.expect(roles.StatementRole, statement) .venv/lib/python3.8/site-packages/sqlalchemy/sql/coercions.py:200: in expect return impl._implicit_coercions( .venv/lib/python3.8/site-packages/sqlalchemy/sql/coercions.py:836: in _implicit_coercions return super(StatementImpl, self)._implicit_coercions( .venv/lib/python3.8/site-packages/sqlalchemy/sql/coercions.py:242: in _implicit_coercions self._raise_for_expected(element, argname, resolved) .venv/lib/python3.8/site-packages/sqlalchemy/sql/coercions.py:270: in _raise_for_expected util.raise_(exc.ArgumentError(msg, code=code), replace_context=err) E sqlalchemy.exc.ArgumentError: Executable SQL or text() construct expected, got <sqlalchemy.sql.selectable.Subquery at 0x7fb34b5e1880; test>. ``` * fix get_columns tests * fix instant defaults listener * remove sort_query and get_query_entities The sort_query and get_query_entities functions never worked fully as intended and contained lots of quirky edge cases: 1. sort_query function was dangerous in a sense that it could be used for really inefficient queries (sorting by non-indexed column). 2. The entity string label introspection in both functions relied on SQLAlchemy internals which were drastically changed in SA 1.4. Relying on those was never a good idea in the first place. * Remove also `get_query_entity_by_alias()` which depends on `get_query_entities()` which was removed in commit 460e1da . * Fix linting errors * fix scalar_subquery warnings * Update sqlalchemy_utils/compat.py Co-authored-by: Nicola Soranzo <nicola.soranzo@gmail.com> * fix selectable columns warning * Add health checks for SQL Server docker service Remove `!` from password so `SA_PASSWORD` doesn't need extra quotes. * Fix install_mssql.sh commands * Use `isolation_level` argument also for `mssql` * Restore `engine.dispose()` * Add `coverage.xml` to `.gitignore` * change tests to use non-deprecated class Change EncryptedType tests to use StringEncryptedType * use scalar subquery for sqlalchemy 1.4 * use create_mock_engine for SA 1.4 * isort linting fixes Co-authored-by: Matthias Bernt <m.bernt@ufz.de> Co-authored-by: Srinivasan R <srinivasanr@gmail.com> Co-authored-by: Luca Venturini <luca.venturini@earlham.ac.uk> Co-authored-by: Braden Mars <bradenmars@bradenmars.me> Co-authored-by: Konsta Vesterinen <konsta.vesterinen@gmail.com>
- Loading branch information
6 people
committed
Apr 12, 2021
1 parent
2e8ee00
commit a4154bd