AttributeError 'NoneType' object has no attribute '_from_objects' persistent exception

[ejlangev]

Describe the bug

We get an unexpected exception when attempting to query our database that seems to put SQLAlchemy into some kind of broken state from which it cannot recover. We are not able to reproduce the issue regularly but when it does happen the instance of the application in question cannot run any queries without generating this exception and will never recover. The queries are fairly innocuous such as:

session.scalar(select(MyRecord).options(joinedload(MyRecord.associated_object).joinedload(AssociatedObject.other_object)).where(MyRecord.attr1 == "some value")

The only pattern we have seen is that it always happens at startup as in the container will exhibit the exception from the very first request it receives and then fail onwards from that point on for all requests using the database and it does not seem that containers that successfully start up and serve requests ever run into this problem after they succeed the first time. This seems to point to some kind of initialization issue or maybe some part of the initialization code not being thread safe.

Sorry there is not more detail here, have been unable to reproduce this despite a lot of attempts

Optional link from https://docs.sqlalchemy.org which documents the behavior that is expected

No response

SQLAlchemy Version in Use

2.0.4

DBAPI (i.e. the database driver)

psycopg2

Database Vendor and Major Version

PostgreSQL 14 (via AlloyDB on GCP)

Python Version

3.10

Operating system

Linux

To Reproduce

We are not able to reproduce this regularly but when it happens it impacts all requests coming into a running instance of our application and never recovers. The exception happens as part of multiple queries so it does not seem to be tied to a specific query.

Error

Exception

AttributeError 'NoneType' object has no attribute '_from_objects'

Stack Trace

sqlalchemy/sql/selectable.py in <listcomp> at line 4625
sqlalchemy/sql/selectable.py in _get_froms at line 4624
sqlalchemy/sql/selectable.py in __init__ at line 4526
sqlalchemy/orm/context.py in create_for_statement at line 1065
sqlalchemy/sql/base.py in create_for_statement at line 650
sqlalchemy/sql/compiler.py in visit_select at line 4406
sqlalchemy/sql/visitors.py in _compiler_dispatch at line 143
sqlalchemy/sql/compiler.py in process at line 755
sqlalchemy/sql/compiler.py in __init__ at line 710
sqlalchemy/sql/compiler.py in __init__ at line 1269
sqlalchemy/sql/elements.py in _compiler at line 290
sqlalchemy/sql/elements.py in _compile_w_cache at line 651
sqlalchemy/engine/base.py in _execute_clauseelement at line 1630
sqlalchemy/sql/elements.py in _execute_on_connection at line 485
sqlalchemy/engine/base.py in execute at line 1414
sqlalchemy/orm/context.py in orm_execute_statement at line 253
sqlalchemy/orm/session.py in _execute_internal at line 2124
sqlalchemy/orm/session.py in scalar at line 2279

Additional context

We are using FastAPI to run our backend.

[zzzeek]

hi -

re: "We are not able to reproduce this regularly but when it happens it impacts all requests coming into a running instance of our application and never recovers." this seems like a race condition of some kind, such as your modules are still being imported when part of the program starts serving requests. The NoneType refers to getting a None into the "_raw_columns" collection of a Select which is not possible without some kind of invalid state elsewhere. We get many, many...many......MANY people using FastAPI who have race-condition related issues. I would ensure all modules are fully imported and registry.configure is called, before any threads/tasks/greenlets/subprocesses start spinning up, however fastAPI has you configure that. I'd reach out to FastAPI for more help.

[ejlangev]

I added a call to configure_mappers from sqlalchemy.orm that happens during start up prior to requests being served. Will see if that helps resolve the issue.

[ejlangev]

The error persists even after calling configure_mappers after importing all our database related code but before any requests are served. Any other suggestions of what we might do to resolve?

[ejlangev]

do you have any event hooks in play? things that try to modify statements as they are created / compiled / executed?

We are using opentelemetry to instrument our application including for sqlalchemy. Here is the code for it: https://github.com/open-telemetry/opentelemetry-python-contrib/blob/main/instrumentation/opentelemetry-instrumentation-sqlalchemy/src/opentelemetry/instrumentation/sqlalchemy/engine.py. We pass our engine to SQLAlchemyInstrumentor().instrument(engine=engine) at module load time.

I don't know 100% what raw_session is but you are doing multiple strange things there:

You can see the definition of it in the same gist. It's just an invocation of a sessionmaker. We are passing the session across thread boundaries now that you mention it. We are opening it in one thread and then offloading the use into another thread. However, the coroutine that creates the session waits for the use of the session in the other thread to be done prior to moving on and the sessions are not shared otherwise. Is sharing a session between threads definitely unsafe?

If you are using asyncio, why are you using psycopg2?

We are using trio rather than asyncio and none of the database drivers supported by sqlalchemy use either trio or anyio unfortunately. This is why we are using trio.to_thread.run_sync which offloads a function call into a thread. This ideally means that we don't block our main thread during the IO that the database driver is doing.

do you have any @compiles recipes in play?

We do not use any of these.

[CaselIT]

do you have any event hooks in play? things that try to modify statements as they are created / compiled / executed?

We are using opentelemetry to instrument our application including for sqlalchemy. Here is the code for it: https://github.com/open-telemetry/opentelemetry-python-contrib/blob/main/instrumentation/opentelemetry-instrumentation-sqlalchemy/src/opentelemetry/instrumentation/sqlalchemy/engine.py. We pass our engine to SQLAlchemyInstrumentor().instrument(engine=engine) at module load time.

I think it would be useful to try removing as much extra dependencies as possible to try to find the minimum set that causes the issue.

You can see the definition of it in the same gist. It's just an invocation of a sessionmaker.

I assumed as much, but the function uses SessionReadReplicaLocal that's not defined in the gist.

We are passing the session across thread boundaries now that you mention it. We are opening it in one thread and then offloading the use into another thread. However, the coroutine that creates the session waits for the use of the session in the other thread to be done prior to moving on and the sessions are not shared otherwise. Is sharing a session between threads definitely unsafe?

It should not be, but I think it's bad practice, since it's easy to forget somewhere and you get concurrent access. If you ensure that does not happen it should not be a problem.

If you are using asyncio, why are you using psycopg2?

We are using trio rather than asyncio and none of the database drivers supported by sqlalchemy use either trio or anyio unfortunately. This is why we are using trio.to_thread.run_sync which offloads a function call into a thread. This ideally means that we don't block our main thread during the IO that the database driver is doing.

well not really, session close may do IO to rollback, and that will block the loop (hopefully not for long).
As a side note I never understood why there are multiple incompatible implementation of asyncio in python..

[ejlangev]

I think it would be useful to try removing as much extra dependencies as possible to try to find the minimum set that causes the issue.

Not a bad idea, I'll remove the instrumentation.

the function uses SessionReadReplicaLocal

It should be called SessionLocal which is defined in there. I updated the gist.

If you ensure that does not happen it should not be a problem.

I'm sure there isn't concurrent access with how it is implemented but it does pass across thread boundaries.

multiple incompatible implementation of asyncio in python

Trio is easier to reason about and has better cancellation semantics than asyncio but isn't in the standard library.

[zzzeek]

do you have any event hooks in play? things that try to modify statements as they are created / compiled / executed?

We are using opentelemetry to instrument our application including for sqlalchemy. Here is the code for it: https://github.com/open-telemetry/opentelemetry-python-contrib/blob/main/instrumentation/opentelemetry-instrumentation-sqlalchemy/src/opentelemetry/instrumentation/sqlalchemy/engine.py. We pass our engine to SQLAlchemyInstrumentor().instrument(engine=engine) at module load time.

OK! So I dont see anything in that file that would do anything like we see here, but we have this situation where you have an "impossible" error, that suggests something going wrong with the state of objects outside of SQLAlchemy or your application, and then it's revealed that this fairly unusual library that seems to be doing things like span.set_attribute(SpanAttributes.DB_STATEMENT, statement), whatever that means, is going on, so this seems very possible as something that could be related.

can you perhaps try running things without this telemetry system plugged in and see if anything changes?

We are using trio rather than asyncio and none of the database drivers supported by sqlalchemy use either trio or anyio unfortunately. This is why we are using trio.to_thread.run_sync which offloads a function call into a thread. This ideally means that we don't block our main thread during the IO that the database driver is doing.

if to_thread is using plain Python Thread then it's unlikely that's the source of this very specific issue as this seems to be the state of a function-local object changing mid stream. Unless there is some combination of whatever opentelemetry is doing with this.

[ejlangev]

I've implemented a bunch of these suggestions. Will see if the issue goes away and report back. Right now I have kept the instrumentation in place to try to narrow down what the actual problem is.

[zzzeek]

same issue reported at https://groups.google.com/g/sqlalchemy/c/uVabm2B_Emw/m/xX1RJjWlBgAJ?utm_medium=email&utm_source=footer

anything look familiar in that message?

[zzzeek]

OK we now have a codepath that explains "'NoneType' object has no attribute '_from_objects'" and you likely have the same issue as the poster.

however, I dont know how to reproduce it. But, one patch that would naively solve this, if you could try it, is this:

diff --git a/lib/sqlalchemy/orm/strategies.py b/lib/sqlalchemy/orm/strategies.py
index 8e06c4f598..2bac1aad48 100644
--- a/lib/sqlalchemy/orm/strategies.py
+++ b/lib/sqlalchemy/orm/strategies.py
@@ -218,10 +218,10 @@ class ColumnLoader(LoaderStrategy):
             if adapter:
                 if check_for_adapt:
                     c = adapter.adapt_check_present(c)
-                    if c is None:
-                        return
                 else:
                     c = adapter.columns[c]
+                if c is None:
+                    return
 
             compile_state._append_dedupe_col_collection(c, column_collection)

[zzzeek]

I've started #9777 where I have a patch that simplifies part of the JoinedLoader, in an area that seems a reasonable place to guess that this issue might be arising from. the patch is the review at https://gerrit.sqlalchemy.org/c/sqlalchemy/sqlalchemy/+/4615 and while I can't pinpoint exactly what's happening, this patch removes a whole knot where a risky thread-sensitive thing was going on and just simplifies it to not do that.