Update for new versions of SQLAlchemy #910

davidism · 2021-02-02T14:59:16Z

SQLAlchemy 1.4 made some changes that are currently incompatible with this extension. Identify failing tests and make changes to fix those. We should also pin to SQLAlchemy<2, since that will be a bigger change and will be more appropriate in Flask-SQLAlchemy 3.0.

Here's some that I'm aware of so far:

URL is immutable from SQLAlchemy 1.4 #885 URL is immutable

The text was updated successfully, but these errors were encountered:

lensonp · 2021-03-15T18:40:15Z

Possibly related. Test suites started failing on db.drop_all() after sqlalchemy 1.3.23->1.4.0.

flask-sqlalchemy 2.4.4
sqlalchemy 1.4.0
pymysql 1.0.2
mysql Ver 8.0.23-0ubuntu0.20.04.1 for Linux on x86_64 ((Ubuntu))
Python 3.8.5
Flask 1.1.2

Stack trace:

tests/test_all.py:23: in <module>
    db.drop_all()
/usr/local/lib/python3.6/dist-packages/flask_sqlalchemy/__init__.py:1047: in drop_all
    self._execute_for_all_tables(app, bind, 'drop_all')
/usr/local/lib/python3.6/dist-packages/flask_sqlalchemy/__init__.py:1031: in _execute_for_all_tables
    op(bind=self.get_engine(app, bind), **extra)
/usr/local/lib/python3.6/dist-packages/flask_sqlalchemy/__init__.py:962: in get_engine
    return connector.get_engine()
/usr/local/lib/python3.6/dist-packages/flask_sqlalchemy/__init__.py:555: in get_engine
    options = self.get_options(sa_url, echo)
/usr/local/lib/python3.6/dist-packages/flask_sqlalchemy/__init__.py:570: in get_options
    self._sa.apply_driver_hacks(self._app, sa_url, options)
/usr/local/lib/python3.6/dist-packages/flask_sqlalchemy/__init__.py:884: in apply_driver_hacks
    sa_url.query.setdefault('charset', 'utf8')
E   AttributeError: 'sqlalchemy.cimmutabledict.immutabledict' object has no attribute 'setdefault'

dpgaspar · 2021-03-15T19:26:01Z

Finding issues also:

  File "/Users/daniel/workarea/preset/Flask-AppBuilder/venv/lib/python3.7/site-packages/Flask_AppBuilder-3.2.0-py3.7.egg/flask_appbuilder/security/sqla/manager.py", line 88, in create_db
    engine = self.get_session.get_bind(mapper=None, clause=None)
  File "<string>", line 2, in get_bind
  File "/Users/daniel/workarea/preset/Flask-AppBuilder/venv/lib/python3.7/site-packages/sqlalchemy/orm/scoping.py", line 104, in _proxied
    return self.registry()
  File "/Users/daniel/workarea/preset/Flask-AppBuilder/venv/lib/python3.7/site-packages/sqlalchemy/util/_collections.py", line 1010, in __call__
    return self.registry.setdefault(key, self.createfunc())
  File "/Users/daniel/workarea/preset/Flask-AppBuilder/venv/lib/python3.7/site-packages/sqlalchemy/orm/session.py", line 4027, in __call__
    return self.class_(**local_kw)
  File "/Users/daniel/workarea/preset/Flask-AppBuilder/venv/lib/python3.7/site-packages/flask_sqlalchemy/__init__.py", line 138, in __init__
    bind = options.pop('bind', None) or db.engine
  File "/Users/daniel/workarea/preset/Flask-AppBuilder/venv/lib/python3.7/site-packages/flask_sqlalchemy/__init__.py", line 943, in engine
    return self.get_engine()
  File "/Users/daniel/workarea/preset/Flask-AppBuilder/venv/lib/python3.7/site-packages/flask_sqlalchemy/__init__.py", line 962, in get_engine
    return connector.get_engine()
  File "/Users/daniel/workarea/preset/Flask-AppBuilder/venv/lib/python3.7/site-packages/flask_sqlalchemy/__init__.py", line 555, in get_engine
    options = self.get_options(sa_url, echo)
  File "/Users/daniel/workarea/preset/Flask-AppBuilder/venv/lib/python3.7/site-packages/flask_sqlalchemy/__init__.py", line 570, in get_options
    self._sa.apply_driver_hacks(self._app, sa_url, options)
  File "/Users/daniel/workarea/preset/Flask-AppBuilder/venv/lib/python3.7/site-packages/flask_sqlalchemy/__init__.py", line 914, in apply_driver_hacks
    sa_url.database = os.path.join(app.root_path, sa_url.database)
AttributeError: can't set attribute

SQLAlchemy 1.4.0 just got released

ThiefMaster · 2021-03-15T19:27:50Z

Just FYI, I would strongly suggest anyone to pin sqlalchemy to <1.4 - 1.x releases in SA are generally major/breaking releases.

brmzkw · 2021-03-15T22:07:43Z

This is breaking with SQLAlchemy 1.4:

from flask import Flask
from flask_sqlalchemy import SQLAlchemy
from sqlalchemy.orm import joinedload

app = Flask(__name__)

app.config['SQLALCHEMY_DATABASE_URI'] = 'sqlite:///:memory:'
app.config['SQLALCHEMY_TRACK_MODIFICATIONS'] = False

db = SQLAlchemy(app)

class User(db.Model):
    id = db.Column(db.Integer, primary_key=True)
    manager_id = db.Column(db.Integer, db.ForeignKey('user.id'))
    manager = db.relationship('User', remote_side=[id])


db.create_all()

u = User()
db.session.add(u)
db.session.commit()

u2 = User()
db.session.add(u2)
db.session.commit()

paginate = db.session.query(User, User.manager).options(joinedload(User.manager)).paginate()

Traceback (most recent call last):
  File "bug-1.4.py", line 28, in <module>
    paginate = db.session.query(User, User.manager).options(joinedload(User.manager)).paginate()
  File "/venv/lib/python3.7/site-packages/flask_sqlalchemy/__init__.py", line 501, in paginate
    total = self.order_by(None).count()
  File "/venv/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3009, in count
    return self._from_self(col).scalar()
  File "/venv/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 2753, in scalar
    ret = self.one()
  File "/venv/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 2730, in one
    return self._iter().one()
  File "/venv/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 2771, in _iter
    execution_options={"_sa_orm_load_options": self.load_options},
  File "/venv/lib/python3.7/site-packages/sqlalchemy/orm/session.py", line 1653, in execute
    result = conn._execute_20(statement, params or {}, execution_options)
  File "/venv/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1520, in _execute_20
    return meth(self, args_10style, kwargs_10style, execution_options)
  File "/venv/lib/python3.7/site-packages/sqlalchemy/sql/elements.py", line 314, in _execute_on_connection
    self, multiparams, params, execution_options
  File "/venv/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1387, in _execute_clauseelement
    linting=self.dialect.compiler_linting | compiler.WARN_LINTING,
  File "/venv/lib/python3.7/site-packages/sqlalchemy/sql/elements.py", line 539, in _compile_w_cache
    **kw
  File "/venv/lib/python3.7/site-packages/sqlalchemy/sql/elements.py", line 566, in _compiler
    return dialect.statement_compiler(dialect, self, **kw)
  File "/venv/lib/python3.7/site-packages/sqlalchemy/sql/compiler.py", line 766, in __init__
    Compiled.__init__(self, dialect, statement, **kwargs)
  File "/venv/lib/python3.7/site-packages/sqlalchemy/sql/compiler.py", line 455, in __init__
    self.string = self.process(self.statement, **compile_kwargs)
  File "/venv/lib/python3.7/site-packages/sqlalchemy/sql/compiler.py", line 490, in process
    return obj._compiler_dispatch(self, **kwargs)
  File "/venv/lib/python3.7/site-packages/sqlalchemy/sql/visitors.py", line 81, in _compiler_dispatch
    return meth(self, **kw)
  File "/venv/lib/python3.7/site-packages/sqlalchemy/sql/compiler.py", line 2965, in visit_select
    select_stmt, self, **kwargs
  File "/venv/lib/python3.7/site-packages/sqlalchemy/sql/base.py", line 492, in create_for_statement
    return klass.create_for_statement(statement, compiler, **kw)
  File "/venv/lib/python3.7/site-packages/sqlalchemy/orm/context.py", line 579, in create_for_statement
    opt.process_compile_state(self)
  File "/venv/lib/python3.7/site-packages/sqlalchemy/orm/strategy_options.py", line 136, in process_compile_state
    self._process(compile_state, not bool(compile_state.current_path))
  File "/venv/lib/python3.7/site-packages/sqlalchemy/orm/strategy_options.py", line 652, in _process
    raiseerr,
  File "/venv/lib/python3.7/site-packages/sqlalchemy/orm/strategy_options.py", line 796, in _bind_loader
    entities, prop, token._parententity, raiseerr
  File "/venv/lib/python3.7/site-packages/sqlalchemy/orm/strategy_options.py", line 882, in _find_entity_prop_comparator
    % (util.clsname_as_plain_name(type(prop)), prop)
sqlalchemy.exc.ArgumentError: Query has only expression-based entities, which do not apply to relationship property "User.manager"

Issue created on SQLAlchemy: sqlalchemy/sqlalchemy#6052

EDIT: problem fixed with sqlalchemy/sqlalchemy@1c5ec1e

ThiefMaster · 2021-03-15T22:09:40Z

Including errors/tracebacks would be useful, and if it's just for people googling their error to find this issue...

pallets-eco/flask-sqlalchemy#910 pallets-eco/flask-sqlalchemy#885 Hot-fix for now.

zzzeek · 2021-03-16T01:16:14Z

for the URL stuff please note url is now immutable, there was unfortunately no reasonable deprecation path for this combined with the fact that I didn't think mutation of URLs was that prevalent anyway. if flask is trying to "fix up" the arguments when the engine is created it would better if they used the event hook to do so, that way you get the arguments that are going straight to dbapi.connect() and you can deal with sqlite3 or whatever directly there. dialect.name will tell you what kind of database you're dealing with and dialect.driver will give you the DBAPI name.

zzzeek · 2021-03-16T22:11:43Z

Here's a trace for part of flask admin, separate project? they should not be accessing private API like that, if they need class regsitry access it would have been nice if we had been tasked to create a public API

  File "/usr/local/bin/airflow", line 26, in <module>
    from airflow.bin.cli import CLIFactory
  File "/usr/local/lib/python3.7/dist-packages/airflow/bin/cli.py", line 80, in <module>
    from airflow.www.app import (cached_app, create_app)
  File "/usr/local/lib/python3.7/dist-packages/airflow/www/app.py", line 38, in <module>
    from airflow.www.blueprints import routes
  File "/usr/local/lib/python3.7/dist-packages/airflow/www/blueprints.py", line 25, in <module>
    from airflow.www import utils as wwwutils
  File "/usr/local/lib/python3.7/dist-packages/airflow/www/utils.py", line 40, in <module>
    import flask_admin.contrib.sqla.filters as sqlafilters
  File "/usr/local/lib/python3.7/dist-packages/flask_admin/contrib/sqla/__init__.py", line 2, in <module>
    from .view import ModelView
  File "/usr/local/lib/python3.7/dist-packages/flask_admin/contrib/sqla/view.py", line 18, in <module>
    from flask_admin.contrib.sqla.tools import is_relationship
  File "/usr/local/lib/python3.7/dist-packages/flask_admin/contrib/sqla/tools.py", line 4, in <module>
    from sqlalchemy.ext.declarative.clsregistry import _class_resolver
ModuleNotFoundError: No module named 'sqlalchemy.ext.declarative.clsregistry'

ThiefMaster · 2021-03-16T22:15:23Z

yes, separate project: https://github.com/flask-admin/flask-admin

davidism · 2021-03-18T16:41:45Z

Fixed the URL issue in #933, as far as I can tell that's the only incompatibility. Will release 2.5 soon.

davidism added this to the 2.4.5 milestone Feb 2, 2021

davidism added the good first issue label Feb 2, 2021

aadel mentioned this issue Feb 27, 2021

Initial Support for SQLAlchemy 1.4 #919

Closed

6 tasks

This was referenced Mar 15, 2021

Superset 0.38, 0.37 (and potentially others) are broken with sqlachemy 1.4.0 release apache/superset#13631

Closed

fix: sqlalchemy 1.4.0 breaking changes dpgaspar/Flask-AppBuilder#1586

Merged

ThiefMaster mentioned this issue Mar 15, 2021

Error on db.create_all with latest SQLAlchemy #925

Closed

brmzkw mentioned this issue Mar 15, 2021

Regression with joinedload() on 1.4 sqlalchemy/sqlalchemy#6052

Closed

JabLuszko added a commit to JabLuszko/OSM-Rocketmap that referenced this issue Mar 15, 2021

SQLAlchemy 1.4.0 released, breaks flask-sqlalchemy

4ef56aa

pallets-eco/flask-sqlalchemy#910 pallets-eco/flask-sqlalchemy#885 Hot-fix for now.

JabLuszko mentioned this issue Mar 15, 2021

SQLAlchemy 1.4.0 released, breaks flask-sqlalchemy cecpk/RocketMAD#205

Merged

6 tasks

davidism mentioned this issue Mar 16, 2021

quickstart documentation does not work #927

Closed

dpgaspar mentioned this issue Mar 16, 2021

fix: bump FAB to 3.2.1, SQLAlchemy fix apache/superset#13642

Merged

6 tasks

zzzeek mentioned this issue Mar 16, 2021

No module named 'sqlalchemy.ext.declarative.clsregistry' on version 1.4.0 sqlalchemy/sqlalchemy#6065

Closed

aprilahijriyan mentioned this issue Mar 17, 2021

[BREAKING CHANGES] downgrade sqlalchemy <1.4.0 zemfrog/zemfrog#87

Closed

davidism closed this as completed Mar 18, 2021

davidism mentioned this issue Mar 18, 2021

Sqlite db.create_all() crash with AttributeError: can't set attribute #938

Closed

Kanchanakkh mentioned this issue Mar 23, 2021

make a change to a version of Flask-SQLAlchemy pwnlandia/mhn#805

Merged

github-actions bot locked as resolved and limited conversation to collaborators Apr 2, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update for new versions of SQLAlchemy #910

Update for new versions of SQLAlchemy #910

davidism commented Feb 2, 2021 •

edited

lensonp commented Mar 15, 2021

dpgaspar commented Mar 15, 2021 •

edited

ThiefMaster commented Mar 15, 2021 •

edited

brmzkw commented Mar 15, 2021 •

edited

ThiefMaster commented Mar 15, 2021

zzzeek commented Mar 16, 2021

zzzeek commented Mar 16, 2021

ThiefMaster commented Mar 16, 2021

davidism commented Mar 18, 2021

Update for new versions of SQLAlchemy #910

Update for new versions of SQLAlchemy #910

Comments

davidism commented Feb 2, 2021 • edited

lensonp commented Mar 15, 2021

dpgaspar commented Mar 15, 2021 • edited

ThiefMaster commented Mar 15, 2021 • edited

brmzkw commented Mar 15, 2021 • edited

ThiefMaster commented Mar 15, 2021

zzzeek commented Mar 16, 2021

zzzeek commented Mar 16, 2021

ThiefMaster commented Mar 16, 2021

davidism commented Mar 18, 2021

davidism commented Feb 2, 2021 •

edited

dpgaspar commented Mar 15, 2021 •

edited

ThiefMaster commented Mar 15, 2021 •

edited

brmzkw commented Mar 15, 2021 •

edited