(2.x)Tasks run outside of a flow using the Prefect Test Harness fail with a 500 internal server error #17531
Description
Bug summary
Utilizing the prefect_test_harness with a task run outside of a flow run context results in a 500 internal server error this only occurs when running on prefect 2, prefect 3 seems to handle this particular case without issue, the task itself also runs fine outside of the test harness and tests using this seem to "work" as well but the test output is polluted with these error making it more difficult to parse.
from prefect import flow, task
from prefect.testing.utilities import prefect_test_harness
@task
def my_task():
return 42
with prefect_test_harness():
assert my_task() == 42Version info
Version: 2.20.16
API version: 0.8.4
Python version: 3.12.0
Git commit: b5047953
Built: Thu, Dec 19, 2024 10:55 AM
OS/Arch: darwin/arm64
Profile: masonsandbox
Server type: cloud
Additional context
stack trace:
15:17:45.313 | INFO | prefect.engine - Created task run 'my_task-78429a92' for task 'my_task'
15:17:45.409 | INFO | Task run 'my_task-78429a92' - Finished in state Completed()
15:17:45.523 | ERROR | prefect.server - Encountered exception in request:
Traceback (most recent call last):
File "/opt/homebrew/Caskroom/miniconda/base/envs/prefect_2.x_test/lib/python3.12/site-packages/prefect/_vendor/starlette/middleware/errors.py", line 168, in __call__
await self.app(scope, receive, _send)
File "/opt/homebrew/Caskroom/miniconda/base/envs/prefect_2.x_test/lib/python3.12/site-packages/prefect/_vendor/starlette/middleware/gzip.py", line 24, in __call__
await responder(scope, receive, send)
File "/opt/homebrew/Caskroom/miniconda/base/envs/prefect_2.x_test/lib/python3.12/site-packages/prefect/_vendor/starlette/middleware/gzip.py", line 44, in __call__
await self.app(scope, receive, self.send_with_gzip)
File "/opt/homebrew/Caskroom/miniconda/base/envs/prefect_2.x_test/lib/python3.12/site-packages/prefect/_vendor/starlette/middleware/exceptions.py", line 62, in __call__
await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
File "/opt/homebrew/Caskroom/miniconda/base/envs/prefect_2.x_test/lib/python3.12/site-packages/prefect/_vendor/starlette/_exception_handler.py", line 62, in wrapped_app
raise exc
File "/opt/homebrew/Caskroom/miniconda/base/envs/prefect_2.x_test/lib/python3.12/site-packages/prefect/_vendor/starlette/_exception_handler.py", line 51, in wrapped_app
await app(scope, receive, sender)
File "/opt/homebrew/Caskroom/miniconda/base/envs/prefect_2.x_test/lib/python3.12/site-packages/prefect/_vendor/fastapi/middleware/asyncexitstack.py", line 20, in __call__
raise e
File "/opt/homebrew/Caskroom/miniconda/base/envs/prefect_2.x_test/lib/python3.12/site-packages/prefect/_vendor/fastapi/middleware/asyncexitstack.py", line 17, in __call__
await self.app(scope, receive, send)
File "/opt/homebrew/Caskroom/miniconda/base/envs/prefect_2.x_test/lib/python3.12/site-packages/prefect/_vendor/starlette/routing.py", line 760, in __call__
await self.middleware_stack(scope, receive, send)
File "/opt/homebrew/Caskroom/miniconda/base/envs/prefect_2.x_test/lib/python3.12/site-packages/prefect/_vendor/starlette/routing.py", line 780, in app
await route.handle(scope, receive, send)
File "/opt/homebrew/Caskroom/miniconda/base/envs/prefect_2.x_test/lib/python3.12/site-packages/prefect/_vendor/starlette/routing.py", line 302, in handle
await self.app(scope, receive, send)
File "/opt/homebrew/Caskroom/miniconda/base/envs/prefect_2.x_test/lib/python3.12/site-packages/prefect/_vendor/starlette/routing.py", line 81, in app
await wrap_app_handling_exceptions(app, request)(scope, receive, send)
File "/opt/homebrew/Caskroom/miniconda/base/envs/prefect_2.x_test/lib/python3.12/site-packages/prefect/_vendor/starlette/_exception_handler.py", line 62, in wrapped_app
raise exc
File "/opt/homebrew/Caskroom/miniconda/base/envs/prefect_2.x_test/lib/python3.12/site-packages/prefect/_vendor/starlette/_exception_handler.py", line 51, in wrapped_app
await app(scope, receive, sender)
File "/opt/homebrew/Caskroom/miniconda/base/envs/prefect_2.x_test/lib/python3.12/site-packages/prefect/_vendor/starlette/routing.py", line 76, in app
response = await func(request)
^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Caskroom/miniconda/base/envs/prefect_2.x_test/lib/python3.12/site-packages/prefect/server/utilities/server.py", line 104, in handle_response_scoped_depends
response = await default_handler(request)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Caskroom/miniconda/base/envs/prefect_2.x_test/lib/python3.12/site-packages/prefect/_vendor/fastapi/routing.py", line 251, in app
raw_response = await run_endpoint_function(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Caskroom/miniconda/base/envs/prefect_2.x_test/lib/python3.12/site-packages/prefect/_vendor/fastapi/routing.py", line 177, in run_endpoint_function
return await dependant.call(**values)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Caskroom/miniconda/base/envs/prefect_2.x_test/lib/python3.12/site-packages/prefect/server/api/logs.py", line 27, in create_logs
await models.logs.create_logs(session=session, logs=batch)
File "/opt/homebrew/Caskroom/miniconda/base/envs/prefect_2.x_test/lib/python3.12/site-packages/prefect/server/database/dependencies.py", line 125, in async_wrapper
return await fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Caskroom/miniconda/base/envs/prefect_2.x_test/lib/python3.12/site-packages/prefect/server/models/logs.py", line 45, in create_logs
await session.execute(db.insert(db.Log).values())
File "/opt/homebrew/Caskroom/miniconda/base/envs/prefect_2.x_test/lib/python3.12/site-packages/sqlalchemy/ext/asyncio/session.py", line 461, in execute
result = await greenlet_spawn(
^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Caskroom/miniconda/base/envs/prefect_2.x_test/lib/python3.12/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 201, in greenlet_spawn
result = context.throw(*sys.exc_info())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Caskroom/miniconda/base/envs/prefect_2.x_test/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 2362, in execute
return self._execute_internal(
^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Caskroom/miniconda/base/envs/prefect_2.x_test/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 2237, in _execute_internal
conn = self._connection_for_bind(bind)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Caskroom/miniconda/base/envs/prefect_2.x_test/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 2106, in _connection_for_bind
return trans._connection_for_bind(engine, execution_options)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "<string>", line 2, in _connection_for_bind
File "/opt/homebrew/Caskroom/miniconda/base/envs/prefect_2.x_test/lib/python3.12/site-packages/sqlalchemy/orm/state_changes.py", line 139, in _go
ret_value = fn(self, *arg, **kw)
^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Caskroom/miniconda/base/envs/prefect_2.x_test/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 1189, in _connection_for_bind
conn = bind.connect()
^^^^^^^^^^^^^^
File "/opt/homebrew/Caskroom/miniconda/base/envs/prefect_2.x_test/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 3278, in connect
return self._connection_cls(self)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Caskroom/miniconda/base/envs/prefect_2.x_test/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 146, in __init__
self._dbapi_connection = engine.raw_connection()
^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Caskroom/miniconda/base/envs/prefect_2.x_test/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 3302, in raw_connection
return self.pool.connect()
^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Caskroom/miniconda/base/envs/prefect_2.x_test/lib/python3.12/site-packages/sqlalchemy/pool/base.py", line 449, in connect
return _ConnectionFairy._checkout(self)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Caskroom/miniconda/base/envs/prefect_2.x_test/lib/python3.12/site-packages/sqlalchemy/pool/base.py", line 1263, in _checkout
fairy = _ConnectionRecord.checkout(pool)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Caskroom/miniconda/base/envs/prefect_2.x_test/lib/python3.12/site-packages/sqlalchemy/pool/base.py", line 712, in checkout
rec = pool._do_get()
^^^^^^^^^^^^^^
File "/opt/homebrew/Caskroom/miniconda/base/envs/prefect_2.x_test/lib/python3.12/site-packages/sqlalchemy/pool/impl.py", line 308, in _do_get
return self._create_connection()
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Caskroom/miniconda/base/envs/prefect_2.x_test/lib/python3.12/site-packages/sqlalchemy/pool/base.py", line 390, in _create_connection
return _ConnectionRecord(self)
^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Caskroom/miniconda/base/envs/prefect_2.x_test/lib/python3.12/site-packages/sqlalchemy/pool/base.py", line 674, in __init__
self.__connect()
File "/opt/homebrew/Caskroom/miniconda/base/envs/prefect_2.x_test/lib/python3.12/site-packages/sqlalchemy/pool/base.py", line 900, in __connect
with util.safe_reraise():
File "/opt/homebrew/Caskroom/miniconda/base/envs/prefect_2.x_test/lib/python3.12/site-packages/sqlalchemy/util/langhelpers.py", line 146, in __exit__
raise exc_value.with_traceback(exc_tb)
File "/opt/homebrew/Caskroom/miniconda/base/envs/prefect_2.x_test/lib/python3.12/site-packages/sqlalchemy/pool/base.py", line 896, in __connect
self.dbapi_connection = connection = pool._invoke_creator(self)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Caskroom/miniconda/base/envs/prefect_2.x_test/lib/python3.12/site-packages/sqlalchemy/engine/create.py", line 643, in connect
return dialect.connect(*cargs, **cparams)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Caskroom/miniconda/base/envs/prefect_2.x_test/lib/python3.12/site-packages/sqlalchemy/engine/default.py", line 621, in connect
return self.loaded_dbapi.connect(*cargs, **cparams)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Caskroom/miniconda/base/envs/prefect_2.x_test/lib/python3.12/site-packages/sqlalchemy/dialects/sqlite/aiosqlite.py", line 352, in connect
await_only(connection),
^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Caskroom/miniconda/base/envs/prefect_2.x_test/lib/python3.12/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 132, in await_only
return current.parent.switch(awaitable) # type: ignore # noqa: E501
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Caskroom/miniconda/base/envs/prefect_2.x_test/lib/python3.12/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 196, in greenlet_spawn
value = await result
^^^^^^^^^^^^
File "/opt/homebrew/Caskroom/miniconda/base/envs/prefect_2.x_test/lib/python3.12/site-packages/aiosqlite/core.py", line 139, in __await__
self.start()
File "/opt/homebrew/Caskroom/miniconda/base/envs/prefect_2.x_test/lib/python3.12/threading.py", line 971, in start
_start_new_thread(self._bootstrap, ())
RuntimeError: can't create new thread at interpreter shutdown
--- Error logging to API ---
Server error '500 Internal Server Error' for url 'http://ephemeral-prefect/api/logs/'
Response: {'exception_message': 'Internal Server Error'}
For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/500%