Upgrade databases and SQLAlchemy #5799

musicinmybrain · 2022-12-18T16:25:50Z

Incompatibilities with recent SQLAlchemy versions are fixed in databases 0.7.0. By requiring databases[sqlite] >=0.7.0,<0.8.0 for testing, we can stop pinning an old SQLAlchemy version, instead requiring sqlalchemy >=1.4.42,<1.5.

Fixes #5556.

github-actions · 2022-12-18T16:30:03Z

📝 Docs preview for commit 574b0a2 at: https://639f3fe2d8175d240d2d6490--fastapi.netlify.app

github-actions · 2023-01-07T22:42:05Z

📝 Docs preview for commit bd2295f at: https://63b9f42d42bbfe7af491a2cf--fastapi.netlify.app

Ryandaydev

Checked out this PR, installed updates, and ran tests.

Test failing:
FAILED tests/test_tutorial/test_async_sql_databases/test_tutorial001.py::test_create_read - sqlalchemy.exc.RemovedIn20Warning: Deprecated API features detected! These feature(s) are not compatib...

sqlalchemy.exc.RemovedIn20Warning: Deprecated API features detected! These feature(s) are not compatible with SQLAlchemy 2.0. To prevent incompatible upgrades prior to updating applications, ensure requirements files are pinned to "sqlalchemy<2.0". Set environment variable SQLALCHEMY_WARN_20=1 to show all deprecation warnings. Set environment variable SQLALCHEMY_SILENCE_UBER_WARNING=1 to silence this message. (Background on SQLAlchemy 2.0 at: https://sqlalche.me/e/b8d9)

musicinmybrain · 2023-01-15T16:37:23Z

Thanks for looking at it. Those warnings were added in sqlalchemy 1.4.46, which was released after I opened this PR. I see three possible courses of action:

Since the version specification in the PR does not permit SQLAlchemy 2.0, ignore these forward-compatibility warnings (by filtering them in pyproject.toml or by setting the environment variable SQLALCHEMY_SILENCE_UBER_WARNING=1 to suppress them).
Pin SQLAlchemy to <1.4.46; but this also keeps users from benefiting from the bugfixes in an otherwise-compatible release.
Find the places where deprecated API features are used, and adjust them to be forward-compatible with SQLAlchemy 2.0.

I’ll take a closer look at this and see if there is an obvious way to do option 3.

musicinmybrain · 2023-01-17T00:22:14Z

Here’s the full output:

====================================================== FAILURES ======================================================
__________________________________________________ test_create_read __________________________________________________

    def test_create_read():
        with TestClient(app) as client:
            note = {"text": "Foo bar", "completed": False}
            response = client.post("/notes/", json=note)
            assert response.status_code == 200, response.text
            data = response.json()
            assert data["text"] == note["text"]
            assert data["completed"] == note["completed"]
            assert "id" in data
>           response = client.get("/notes/")

tests/test_tutorial/test_async_sql_databases/test_tutorial001.py:129: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
_e/lib64/python3.11/site-packages/starlette/testclient.py:488: in get
    return super().get(
_e/lib64/python3.11/site-packages/httpx/_client.py:1045: in get
    return self.request(
_e/lib64/python3.11/site-packages/starlette/testclient.py:454: in request
    return super().request(
_e/lib64/python3.11/site-packages/httpx/_client.py:821: in request
    return self.send(request, auth=auth, follow_redirects=follow_redirects)
_e/lib64/python3.11/site-packages/httpx/_client.py:908: in send
    response = self._send_handling_auth(
_e/lib64/python3.11/site-packages/httpx/_client.py:936: in _send_handling_auth
    response = self._send_handling_redirects(
_e/lib64/python3.11/site-packages/httpx/_client.py:973: in _send_handling_redirects
    response = self._send_single_request(request)
_e/lib64/python3.11/site-packages/httpx/_client.py:1009: in _send_single_request
    response = transport.handle_request(request)
_e/lib64/python3.11/site-packages/starlette/testclient.py:337: in handle_request
    raise exc
_e/lib64/python3.11/site-packages/starlette/testclient.py:334: in handle_request
    portal.call(self.app, scope, receive, send)
_e/lib64/python3.11/site-packages/anyio/from_thread.py:283: in call
    return cast(T_Retval, self.start_task_soon(func, *args).result())
/usr/lib64/python3.11/concurrent/futures/_base.py:456: in result
    return self.__get_result()
/usr/lib64/python3.11/concurrent/futures/_base.py:401: in __get_result
    raise self._exception
_e/lib64/python3.11/site-packages/anyio/from_thread.py:219: in _call_func
    retval = await retval
fastapi/applications.py:270: in __call__
    await super().__call__(scope, receive, send)
_e/lib64/python3.11/site-packages/starlette/applications.py:124: in __call__
    await self.middleware_stack(scope, receive, send)
_e/lib64/python3.11/site-packages/starlette/middleware/errors.py:184: in __call__
    raise exc
_e/lib64/python3.11/site-packages/starlette/middleware/errors.py:162: in __call__
    await self.app(scope, receive, _send)
_e/lib64/python3.11/site-packages/starlette/middleware/exceptions.py:79: in __call__
    raise exc
_e/lib64/python3.11/site-packages/starlette/middleware/exceptions.py:68: in __call__
    await self.app(scope, receive, sender)
fastapi/middleware/asyncexitstack.py:21: in __call__
    raise e
fastapi/middleware/asyncexitstack.py:18: in __call__
    await self.app(scope, receive, send)
_e/lib64/python3.11/site-packages/starlette/routing.py:706: in __call__
    await route.handle(scope, receive, send)
_e/lib64/python3.11/site-packages/starlette/routing.py:276: in handle
    await self.app(scope, receive, send)
_e/lib64/python3.11/site-packages/starlette/routing.py:66: in app
    response = await func(request)
fastapi/routing.py:255: in app
    content = await serialize_response(
fastapi/routing.py:131: in serialize_response
    value, errors_ = field.validate(response_content, {}, loc=("response",))
pydantic/fields.py:895: in pydantic.fields.ModelField.validate
    ???
pydantic/fields.py:928: in pydantic.fields.ModelField._validate_sequence_like
    ???
pydantic/fields.py:1094: in pydantic.fields.ModelField._validate_singleton
    ???
pydantic/fields.py:884: in pydantic.fields.ModelField.validate
    ???
pydantic/fields.py:1101: in pydantic.fields.ModelField._validate_singleton
    ???
pydantic/fields.py:1148: in pydantic.fields.ModelField._apply_validators
    ???
pydantic/class_validators.py:318: in pydantic.class_validators._generic_validator_basic.lambda13
    ???
pydantic/main.py:717: in pydantic.main.BaseModel.validate
    ???
<string>:2: in keys
    ???
_e/lib64/python3.11/site-packages/sqlalchemy/util/deprecations.py:467: in warned
    _warn_with_version(message, version, wtype, stacklevel=3)
_e/lib64/python3.11/site-packages/sqlalchemy/util/deprecations.py:47: in _warn_with_version
    _emit_uber_warning(type_, stacklevel)
_e/lib64/python3.11/site-packages/sqlalchemy/util/deprecations.py:105: in _emit_uber_warning
    _warnings_warn(warn, stacklevel=stacklevel + 1)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

message = RemovedIn20Warning('Deprecated API features detected! These feature(s) are not compatible with SQLAlchemy 2.0. To prev...o show all deprecation warnings.  Set environment variable SQLALCHEMY_SILENCE_UBER_WARNING=1 to silence this message.')
category = None, stacklevel = 5

    def _warnings_warn(message, category=None, stacklevel=2):
    
        # adjust the given stacklevel to be outside of SQLAlchemy
        try:
            frame = sys._getframe(stacklevel)
        except ValueError:
            # being called from less than 3 (or given) stacklevels, weird,
            # but don't crash
            stacklevel = 0
        except:
            # _getframe() doesn't work, weird interpreter issue, weird,
            # ok, but don't crash
            stacklevel = 0
        else:
            # using __name__ here requires that we have __name__ in the
            # __globals__ of the decorated string functions we make also.
            # we generate this using {"__name__": fn.__module__}
            while frame is not None and re.match(
                r"^(?:sqlalchemy\.|alembic\.)", frame.f_globals.get("__name__", "")
            ):
                frame = frame.f_back
                stacklevel += 1
    
        if category is not None:
            warnings.warn(message, category, stacklevel=stacklevel + 1)
        else:
>           warnings.warn(message, stacklevel=stacklevel + 1)
E           sqlalchemy.exc.RemovedIn20Warning: Deprecated API features detected! These feature(s) are not compatible with SQLAlchemy 2.0. To prevent incompatible upgrades prior to updating applications, ensure requirements files are pinned to "sqlalchemy<2.0". Set environment variable SQLALCHEMY_WARN_20=1 to show all deprecation warnings.  Set environment variable SQLALCHEMY_SILENCE_UBER_WARNING=1 to silence this message. (Background on SQLAlchemy 2.0 at: https://sqlalche.me/e/b8d9)

_e/lib64/python3.11/site-packages/sqlalchemy/util/langhelpers.py:1679: RemovedIn20Warning
============================================== short test summary info ===============================================
FAILED tests/test_tutorial/test_async_sql_databases/test_tutorial001.py::test_create_read - sqlalchemy.exc.RemovedIn20Warning: Deprecated API features detected! These feature(s) are not compatible with SQL...
===================================== 1 failed, 1400 passed in 90.00s (0:01:30) ======================================

Unfortunately, this doesn’t carry any information about which deprecated API feature was used, and it’s hard to even be sure where the deprecated usage is happening.

I tried setting 'always::DeprecationWarning', in tool.pytest.ini_options.filterwarnings in pyproject.toml, and got this, which is much briefer but more helpful:

tests/test_tutorial/test_async_sql_databases/test_tutorial001.py::test_create_read
  /home/ben/src/forks/fastapi/fastapi/routing.py:131: RemovedIn20Warning: Deprecated API features detected! These feature(s) are not compatible with SQLAlchemy 2.0. To prevent incompatible upgrades prior to updating applications, ensure requirements files are pinned to "sqlalchemy<2.0". Set environment variable SQLALCHEMY_WARN_20=1 to show all deprecation warnings.  Set environment variable SQLALCHEMY_SILENCE_UBER_WARNING=1 to silence this message. (Background on SQLAlchemy 2.0 at: https://sqlalche.me/e/b8d9)
    value, errors_ = field.validate(response_content, {}, loc=("response",))

That points at an incompatibility in FastAPI, and it is something specific enough I can reasonably filter it out.

musicinmybrain · 2023-01-17T00:23:11Z

Added 'ignore::sqlalchemy.exc.RemovedIn20Warning:fastapi', to tool.pytest.ini_options.filterwarnings in pyproject.toml, along with a comment justifying the filtering.

github-actions · 2023-01-17T00:26:56Z

📝 Docs preview for commit 2deb233 at: https://63c5eb1e4a9d912fcb4f5299--fastapi.netlify.app

Ryandaydev

After warning suppressed, tests are passing.

github-actions · 2023-03-04T18:24:32Z

📝 Docs preview for commit 656ce50 at: https://64038cb82bd72c60f5c3e0c2--fastapi.netlify.app

musicinmybrain · 2023-03-11T14:51:39Z

Rebased on master again.

github-actions · 2023-03-11T14:55:23Z

📝 Docs preview for commit 262fa51 at: https://640c9640611f6123320901d1--fastapi.netlify.app

musicinmybrain · 2023-05-27T16:19:23Z

Rebased on master again.

github-actions · 2023-05-27T16:24:36Z

📝 Docs preview for commit c4a4592 at: https://64722e9cae27a515156fe6e0--fastapi.netlify.app

musicinmybrain · 2023-06-21T14:50:22Z

Rebased on master again.

tiangolo · 2023-06-21T14:55:42Z

📝 Docs preview for commit 3017463 at: https://64930f371649720e5435aa03--fastapi.netlify.app

musicinmybrain · 2023-07-12T20:28:09Z

Rebased again.

musicinmybrain · 2023-08-09T18:58:56Z

@tiangolo, any interest in merging this? I’ve been carrying it as a downstream patch in Fedora Linux for quite a while, and it seems like a straighforward improvement that would allow CI testing with current SQLAlchemy releases in the 1.4.x series.

musicinmybrain · 2023-09-12T11:10:42Z

Rebased again, and adjusted to allow databases 0.8.x too (databases[sqlite] >=0.7.0,<0.9.0).

jonocodes · 2023-10-15T01:44:56Z

Can you clarify. Is SqlAlchemy 2.0 not supported yet? Or is that just a limitation of testing?

musicinmybrain · 2023-10-15T03:12:05Z

Can you clarify. Is SqlAlchemy 2.0 not supported yet? Or is that just a limitation of testing?

These are only test dependencies.

The actual FastAPI package doesn’t import sqlalchemy, but some of the tests (and examples in the documentation, which can be run as tests) do. The databases usage is only in docs_src/async_sql_databases/tutorial001.py, but that means databases constrains the version of sqlalchemy used for testing.

This PR predates the SQLAlchemy 2.0.0 release, so updating to databases 0.7.0 allowed tests to run on the then-latest version of sqlalchemy. Now databases is at 0.8.0, which needs sqlalchemy>=1.4.42,<1.5. So databases is again blocking testing with the latest version of SQLAlchemy, and I should probably edit this PR to restore the original comment about removing the databases dependency in order to upgrade SQLAlchemy.

Still, even if it wouldn’t allow testing with SQLAlchemy 2.0, this PR would at least improve the status quo.

musicinmybrain · 2023-10-15T12:59:20Z

[…] I should probably edit this PR to restore the original comment about removing the databases dependency in order to upgrade SQLAlchemy.

Rebased on master, and the PR no longer removes the mentioned comment.

musicinmybrain · 2023-10-15T13:06:05Z

Added 'ignore::sqlalchemy.exc.RemovedIn20Warning:fastapi', to tool.pytest.ini_options.filterwarnings in pyproject.toml, along with a comment justifying the filtering.

It looks like this is no longer necessary in the current version of FastAPI, so I dropped the second commit.

For posterity, since I’m force-pushing again, that commit looked like:

From 6f11a91c4c60914075a5ca36594be499c524f71a Mon Sep 17 00:00:00 2001
From: "Benjamin A. Beasley" <code@musicinmybrain.net>
Date: Mon, 16 Jan 2023 19:19:56 -0500
Subject: [PATCH] Ignore SQLAlchemy RemovedIn20Warning from FastAPI

We have SQLAlchemy pinned anyway; we are not yet concerned about
forward-compatibility with 2.0, especially in dependencies.
---
 pyproject.toml | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/pyproject.toml b/pyproject.toml
index 2870b31a..4aa9af44 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -97,6 +97,9 @@ filterwarnings = [
     "error",
     # TODO: needed by asyncio in Python 3.9.7 https://bugs.python.org/issue45097, try to remove on 3.9.8
     'ignore:The loop argument is deprecated since Python 3\.8, and scheduled for removal in Python 3\.10:DeprecationWarning:asyncio',
+    # We have SQLAlchemy pinned anyway; we are not yet concerned about
+    # forward-compatibility with 2.0, especially in dependencies.
+    'ignore::sqlalchemy.exc.RemovedIn20Warning:fastapi',
     'ignore:starlette.middleware.wsgi is deprecated and will be removed in a future release\..*:DeprecationWarning:starlette',
     # TODO: remove after upgrading HTTPX to a version newer than 0.23.0
     # Including PR: https://github.com/encode/httpx/pull/2309
-- 
2.41.0

musicinmybrain · 2023-10-15T13:11:43Z

It looks like this is no longer necessary in the current version of FastAPI, so I dropped the second commit.

Hmm, the warning doesn’t show up in local testing, but still appears in the pydantic-v1 CI environments. I’ll restore the second commit to suppress the warning.

musicinmybrain · 2024-02-08T12:42:49Z

Rebased on master again; added 'ignore::sqlalchemy.exc.RemovedIn20Warning:pydantic' since these warnings now arise from Pydantic as well.

Incompatibilities with recent SQLAlchemy versions are fixed in databases 0.7.0.

We have SQLAlchemy pinned anyway; we are not yet concerned about forward-compatibility with 2.0, especially in dependencies.

musicinmybrain · 2024-04-30T12:49:22Z

Rebased on master again.

tiangolo · 2024-04-30T13:00:00Z

📝 Docs preview for commit dca742e at: https://e0ca5caf.fastapitiangolo.pages.dev

musicinmybrain · 2024-04-30T13:53:46Z

Now upgraded to allow current releases of databases and SQLAlchemy (0.9.x and 2.x, respectively).

Tests are still passing for me locally; we will see if the CI agrees.

musicinmybrain · 2024-04-30T14:04:22Z

Trying to adjust the warning configuration so that this still works with Pydantic v1…

tiangolo · 2024-04-30T14:15:25Z

📝 Docs preview for commit 229748d at: https://b684d8ee.fastapitiangolo.pages.dev

tiangolo · 2024-04-30T15:02:06Z

📝 Docs preview for commit 25fe108 at: https://63296ba9.fastapitiangolo.pages.dev

musicinmybrain · 2024-04-30T15:06:32Z

Now I’m seeing issues like:

E       AssertionError: {"detail":"Email already registered"}
E       assert 400 == 200
E        +  where 400 = <Response [400 Bad Request]>.status_code

and

sqlite3.OperationalError: attempt to write a readonly database

with Pydantic v1. I am not sure what to do about that. Suggestions welcome.

This is still useful as a downstream patch in Fedora, where we have Pydantic v2 only and would like to test with current versions of everything.

musicinmybrain force-pushed the sqlalchemy-databases branch from 574b0a2 to bd2295f Compare January 7, 2023 22:32

Ryandaydev reviewed Jan 10, 2023

View reviewed changes

musicinmybrain force-pushed the sqlalchemy-databases branch from bd2295f to 2deb233 Compare January 17, 2023 00:22

Ryandaydev approved these changes Jan 17, 2023

View reviewed changes

musicinmybrain force-pushed the sqlalchemy-databases branch from 2deb233 to 656ce50 Compare March 4, 2023 18:20

musicinmybrain force-pushed the sqlalchemy-databases branch from 656ce50 to 262fa51 Compare March 11, 2023 14:51

musicinmybrain force-pushed the sqlalchemy-databases branch from 262fa51 to c4a4592 Compare May 27, 2023 16:19

musicinmybrain force-pushed the sqlalchemy-databases branch from c4a4592 to 3017463 Compare June 21, 2023 14:50

musicinmybrain force-pushed the sqlalchemy-databases branch 3 times, most recently from 7c97a7d to 2be88d5 Compare July 1, 2023 22:13

musicinmybrain force-pushed the sqlalchemy-databases branch 2 times, most recently from c08d33d to 7bc2ffb Compare July 12, 2023 20:27

musicinmybrain force-pushed the sqlalchemy-databases branch from 7bc2ffb to feab74c Compare August 26, 2023 13:18

musicinmybrain force-pushed the sqlalchemy-databases branch from feab74c to f212102 Compare September 12, 2023 11:09

tiangolo added the internal label Oct 2, 2023

musicinmybrain force-pushed the sqlalchemy-databases branch from f212102 to 6f11a91 Compare October 15, 2023 12:58

musicinmybrain force-pushed the sqlalchemy-databases branch from 6f11a91 to 6bccc4d Compare October 15, 2023 13:06

musicinmybrain force-pushed the sqlalchemy-databases branch 2 times, most recently from 260844a to 03cb89b Compare February 8, 2024 12:37

musicinmybrain mentioned this pull request Feb 15, 2024

⬆ Update databases[sqlite] requirement from <0.7.0,>=0.3.2 to >=0.3.2,<0.9.0 #10168

Open

musicinmybrain added 2 commits April 30, 2024 08:48

Upgrade databases and SQLAlchemy

077726e

Incompatibilities with recent SQLAlchemy versions are fixed in databases 0.7.0.

Ignore SQLAlchemy RemovedIn20Warning from FastAPI and Pydandic

dca742e

We have SQLAlchemy pinned anyway; we are not yet concerned about forward-compatibility with 2.0, especially in dependencies.

musicinmybrain force-pushed the sqlalchemy-databases branch from 03cb89b to dca742e Compare April 30, 2024 12:48

musicinmybrain force-pushed the sqlalchemy-databases branch from 84ee2f1 to 229748d Compare April 30, 2024 14:08

Allow databases 0.9.x and SQLAlchemy 2.x for testing

25fe108

musicinmybrain force-pushed the sqlalchemy-databases branch from 229748d to 25fe108 Compare April 30, 2024 14:54

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Upgrade databases and SQLAlchemy #5799

Upgrade databases and SQLAlchemy #5799

musicinmybrain commented Dec 18, 2022 •

edited

github-actions bot commented Dec 18, 2022

github-actions bot commented Jan 7, 2023

Ryandaydev left a comment •

edited

musicinmybrain commented Jan 15, 2023

musicinmybrain commented Jan 17, 2023

musicinmybrain commented Jan 17, 2023

github-actions bot commented Jan 17, 2023

Ryandaydev left a comment

github-actions bot commented Mar 4, 2023

musicinmybrain commented Mar 11, 2023

github-actions bot commented Mar 11, 2023

musicinmybrain commented May 27, 2023

github-actions bot commented May 27, 2023

musicinmybrain commented Jun 21, 2023

tiangolo commented Jun 21, 2023

musicinmybrain commented Jul 12, 2023

musicinmybrain commented Aug 9, 2023

musicinmybrain commented Sep 12, 2023

jonocodes commented Oct 15, 2023

musicinmybrain commented Oct 15, 2023

musicinmybrain commented Oct 15, 2023

musicinmybrain commented Oct 15, 2023

musicinmybrain commented Oct 15, 2023

musicinmybrain commented Feb 8, 2024

musicinmybrain commented Apr 30, 2024

tiangolo commented Apr 30, 2024

musicinmybrain commented Apr 30, 2024

musicinmybrain commented Apr 30, 2024

tiangolo commented Apr 30, 2024

tiangolo commented Apr 30, 2024

musicinmybrain commented Apr 30, 2024

Upgrade databases and SQLAlchemy #5799

Are you sure you want to change the base?

Upgrade databases and SQLAlchemy #5799

Conversation

musicinmybrain commented Dec 18, 2022 • edited

github-actions bot commented Dec 18, 2022

github-actions bot commented Jan 7, 2023

Ryandaydev left a comment • edited

Choose a reason for hiding this comment

musicinmybrain commented Jan 15, 2023

musicinmybrain commented Jan 17, 2023

musicinmybrain commented Jan 17, 2023

github-actions bot commented Jan 17, 2023

Ryandaydev left a comment

Choose a reason for hiding this comment

github-actions bot commented Mar 4, 2023

musicinmybrain commented Mar 11, 2023

github-actions bot commented Mar 11, 2023

musicinmybrain commented May 27, 2023

github-actions bot commented May 27, 2023

musicinmybrain commented Jun 21, 2023

tiangolo commented Jun 21, 2023

musicinmybrain commented Jul 12, 2023

musicinmybrain commented Aug 9, 2023

musicinmybrain commented Sep 12, 2023

jonocodes commented Oct 15, 2023

musicinmybrain commented Oct 15, 2023

musicinmybrain commented Oct 15, 2023

musicinmybrain commented Oct 15, 2023

musicinmybrain commented Oct 15, 2023

musicinmybrain commented Feb 8, 2024

musicinmybrain commented Apr 30, 2024

tiangolo commented Apr 30, 2024

musicinmybrain commented Apr 30, 2024

musicinmybrain commented Apr 30, 2024

tiangolo commented Apr 30, 2024

tiangolo commented Apr 30, 2024

musicinmybrain commented Apr 30, 2024

musicinmybrain commented Dec 18, 2022 •

edited

Ryandaydev left a comment •

edited