Replies: 5 comments 10 replies
|
This are dependencies "azure-ai-formrecognizer", |
|
Can you share a bit more about where this is failing in your application? A full traceback and, if possible, a minimal reproducer (a small piece of code that is triggering the error) would help. |
|
I also had the same bug when calling the service /usr/local/lib/python3.10/site-packages/requests/__init__.py:113: RequestsDependencyWarning: urllib3 (2.7.0) or chardet (7.6.0)/charset_normalizer (3.5.1) doesn't match a supported version!
warnings.warn(
Registry store URI not provided. Using backend store URI.
2026/09/03 09:24:57 INFO mlflow.store.db.utils: Creating initial MLflow database tables...
2026/09/03 09:24:57 INFO mlflow.store.db.utils: Updating database tables
2026/09/03 09:24:58 INFO: Uvicorn running on http://0.0.0.0:5006 (Press CTRL+C to quit)
2026/09/03 09:24:58 INFO: Started parent process [41]
/usr/local/lib/python3.10/site-packages/requests/__init__.py:113: RequestsDependencyWarning: urllib3 (2.7.0) or chardet (7.6.0)/charset_normalizer (3.5.1) doesn't match a supported version!
warnings.warn(
/usr/local/lib/python3.10/site-packages/requests/__init__.py:113: RequestsDependencyWarning: urllib3 (2.7.0) or chardet (7.6.0)/charset_normalizer (3.5.1) doesn't match a supported version!
warnings.warn(
/usr/local/lib/python3.10/site-packages/requests/__init__.py:113: RequestsDependencyWarning: urllib3 (2.7.0) or chardet (7.6.0)/charset_normalizer (3.5.1) doesn't match a supported version!
warnings.warn(
/usr/local/lib/python3.10/site-packages/requests/__init__.py:113: RequestsDependencyWarning: urllib3 (2.7.0) or chardet (7.6.0)/charset_normalizer (3.5.1) doesn't match a supported version!
warnings.warn(
/usr/local/lib/python3.10/site-packages/requests/__init__.py:113: RequestsDependencyWarning: urllib3 (2.7.0) or chardet (7.6.0)/charset_normalizer (3.5.1) doesn't match a supported version!
warnings.warn(
/usr/local/lib/python3.10/site-packages/mlflow/server/fastapi_app.py:19: StarletteDeprecationWarning: starlette.middleware.wsgi is deprecated and will be removed in a future release. Please refer to https://github.com/abersheeran/a2wsgi as a replacement.
from starlette.middleware.wsgi import WSGIResponder, build_environ
/usr/local/lib/python3.10/site-packages/mlflow/server/fastapi_app.py:19: StarletteDeprecationWarning: starlette.middleware.wsgi is deprecated and will be removed in a future release. Please refer to https://github.com/abersheeran/a2wsgi as a replacement.
from starlette.middleware.wsgi import WSGIResponder, build_environ
2026/09/03 09:25:01 INFO: Started server process [47]
2026/09/03 09:25:01 INFO: Waiting for application startup.
2026/09/03 09:25:01 INFO: Application startup complete.
/usr/local/lib/python3.10/site-packages/mlflow/server/fastapi_app.py:19: StarletteDeprecationWarning: starlette.middleware.wsgi is deprecated and will be removed in a future release. Please refer to https://github.com/abersheeran/a2wsgi as a replacement.
from starlette.middleware.wsgi import WSGIResponder, build_environ
/usr/local/lib/python3.10/site-packages/mlflow/server/fastapi_app.py:19: StarletteDeprecationWarning: starlette.middleware.wsgi is deprecated and will be removed in a future release. Please refer to https://github.com/abersheeran/a2wsgi as a replacement.
from starlette.middleware.wsgi import WSGIResponder, build_environ
/usr/local/lib/python3.10/site-packages/requests/__init__.py:113: RequestsDependencyWarning: urllib3 (2.7.0) or chardet (7.6.0)/charset_normalizer (3.5.1) doesn't match a supported version!
warnings.warn(
/usr/local/lib/python3.10/site-packages/requests/__init__.py:113: RequestsDependencyWarning: urllib3 (2.7.0) or chardet (7.6.0)/charset_normalizer (3.5.1) doesn't match a supported version!
warnings.warn(
2026/09/03 09:25:01 INFO: Started server process [45]
2026/09/03 09:25:01 INFO: Waiting for application startup.
2026/09/03 09:25:01 INFO: Application startup complete.
/usr/local/lib/python3.10/site-packages/requests/__init__.py:113: RequestsDependencyWarning: urllib3 (2.7.0) or chardet (7.6.0)/charset_normalizer (3.5.1) doesn't match a supported version!
warnings.warn(
/usr/local/lib/python3.10/site-packages/requests/__init__.py:113: RequestsDependencyWarning: urllib3 (2.7.0) or chardet (7.6.0)/charset_normalizer (3.5.1) doesn't match a supported version!
warnings.warn(
/usr/local/lib/python3.10/site-packages/requests/__init__.py:113: RequestsDependencyWarning: urllib3 (2.7.0) or chardet (7.6.0)/charset_normalizer (3.5.1) doesn't match a supported version!
warnings.warn(
2026/09/03 09:25:02 INFO: Started server process [44]
2026/09/03 09:25:02 INFO: Waiting for application startup.
2026/09/03 09:25:02 INFO: Application startup complete.
/usr/local/lib/python3.10/site-packages/requests/__init__.py:113: RequestsDependencyWarning: urllib3 (2.7.0) or chardet (7.6.0)/charset_normalizer (3.5.1) doesn't match a supported version!
warnings.warn(
/usr/local/lib/python3.10/site-packages/requests/__init__.py:113: RequestsDependencyWarning: urllib3 (2.7.0) or chardet (7.6.0)/charset_normalizer (3.5.1) doesn't match a supported version!
warnings.warn(
2026/09/03 09:25:02 INFO: Started server process [46]
2026/09/03 09:25:02 INFO: Waiting for application startup.
2026/09/03 09:25:02 INFO: Application startup complete.
2026/09/03 09:25:04 INFO mlflow.server.jobs.utils: Registered online_scoring_scheduler periodic task (runs every 1 minute)
2026/09/03 09:25:04 INFO mlflow.server.jobs.utils: Registered trace_archival_scheduler periodic task (polls every 1 minute and no-ops when trace archival is disabled or unconfigured)
root@do-ki-team-worker:/home/jenkins/workspace# docker exec -it 1-gen-ai-backend sh
sh-5.2# curl http://1-mlflow:5006
Internal Server Errorsh-5.2# curl http://1-mlflow_server:5006
Invalid Host header - possible DNS rebinding attack detectedsh-5.2# ^C
sh-5.2# exit
exit
root@do-ki-team-worker:/home/jenkins/workspace# docker logs 1-mlflow_server
/usr/local/lib/python3.10/site-packages/requests/__init__.py:113: RequestsDependencyWarning: urllib3 (2.7.0) or chardet (7.6.0)/charset_normalizer (3.5.1) doesn't match a supported version!
warnings.warn(
Registry store URI not provided. Using backend store URI.
2026/09/03 09:24:57 INFO mlflow.store.db.utils: Creating initial MLflow database tables...
2026/09/03 09:24:57 INFO mlflow.store.db.utils: Updating database tables
2026/09/03 09:24:58 INFO: Uvicorn running on http://0.0.0.0:5006 (Press CTRL+C to quit)
2026/09/03 09:24:58 INFO: Started parent process [41]
/usr/local/lib/python3.10/site-packages/requests/__init__.py:113: RequestsDependencyWarning: urllib3 (2.7.0) or chardet (7.6.0)/charset_normalizer (3.5.1) doesn't match a supported version!
warnings.warn(
/usr/local/lib/python3.10/site-packages/requests/__init__.py:113: RequestsDependencyWarning: urllib3 (2.7.0) or chardet (7.6.0)/charset_normalizer (3.5.1) doesn't match a supported version!
warnings.warn(
/usr/local/lib/python3.10/site-packages/requests/__init__.py:113: RequestsDependencyWarning: urllib3 (2.7.0) or chardet (7.6.0)/charset_normalizer (3.5.1) doesn't match a supported version!
warnings.warn(
/usr/local/lib/python3.10/site-packages/requests/__init__.py:113: RequestsDependencyWarning: urllib3 (2.7.0) or chardet (7.6.0)/charset_normalizer (3.5.1) doesn't match a supported version!
warnings.warn(
/usr/local/lib/python3.10/site-packages/requests/__init__.py:113: RequestsDependencyWarning: urllib3 (2.7.0) or chardet (7.6.0)/charset_normalizer (3.5.1) doesn't match a supported version!
warnings.warn(
/usr/local/lib/python3.10/site-packages/mlflow/server/fastapi_app.py:19: StarletteDeprecationWarning: starlette.middleware.wsgi is deprecated and will be removed in a future release. Please refer to https://github.com/abersheeran/a2wsgi as a replacement.
from starlette.middleware.wsgi import WSGIResponder, build_environ
/usr/local/lib/python3.10/site-packages/mlflow/server/fastapi_app.py:19: StarletteDeprecationWarning: starlette.middleware.wsgi is deprecated and will be removed in a future release. Please refer to https://github.com/abersheeran/a2wsgi as a replacement.
from starlette.middleware.wsgi import WSGIResponder, build_environ
2026/09/03 09:25:01 INFO: Started server process [47]
2026/09/03 09:25:01 INFO: Waiting for application startup.
2026/09/03 09:25:01 INFO: Application startup complete.
/usr/local/lib/python3.10/site-packages/mlflow/server/fastapi_app.py:19: StarletteDeprecationWarning: starlette.middleware.wsgi is deprecated and will be removed in a future release. Please refer to https://github.com/abersheeran/a2wsgi as a replacement.
from starlette.middleware.wsgi import WSGIResponder, build_environ
/usr/local/lib/python3.10/site-packages/mlflow/server/fastapi_app.py:19: StarletteDeprecationWarning: starlette.middleware.wsgi is deprecated and will be removed in a future release. Please refer to https://github.com/abersheeran/a2wsgi as a replacement.
from starlette.middleware.wsgi import WSGIResponder, build_environ
/usr/local/lib/python3.10/site-packages/requests/__init__.py:113: RequestsDependencyWarning: urllib3 (2.7.0) or chardet (7.6.0)/charset_normalizer (3.5.1) doesn't match a supported version!
warnings.warn(
/usr/local/lib/python3.10/site-packages/requests/__init__.py:113: RequestsDependencyWarning: urllib3 (2.7.0) or chardet (7.6.0)/charset_normalizer (3.5.1) doesn't match a supported version!
warnings.warn(
2026/09/03 09:25:01 INFO: Started server process [45]
2026/09/03 09:25:01 INFO: Waiting for application startup.
2026/09/03 09:25:01 INFO: Application startup complete.
/usr/local/lib/python3.10/site-packages/requests/__init__.py:113: RequestsDependencyWarning: urllib3 (2.7.0) or chardet (7.6.0)/charset_normalizer (3.5.1) doesn't match a supported version!
warnings.warn(
/usr/local/lib/python3.10/site-packages/requests/__init__.py:113: RequestsDependencyWarning: urllib3 (2.7.0) or chardet (7.6.0)/charset_normalizer (3.5.1) doesn't match a supported version!
warnings.warn(
/usr/local/lib/python3.10/site-packages/requests/__init__.py:113: RequestsDependencyWarning: urllib3 (2.7.0) or chardet (7.6.0)/charset_normalizer (3.5.1) doesn't match a supported version!
warnings.warn(
2026/09/03 09:25:02 INFO: Started server process [44]
2026/09/03 09:25:02 INFO: Waiting for application startup.
2026/09/03 09:25:02 INFO: Application startup complete.
/usr/local/lib/python3.10/site-packages/requests/__init__.py:113: RequestsDependencyWarning: urllib3 (2.7.0) or chardet (7.6.0)/charset_normalizer (3.5.1) doesn't match a supported version!
warnings.warn(
/usr/local/lib/python3.10/site-packages/requests/__init__.py:113: RequestsDependencyWarning: urllib3 (2.7.0) or chardet (7.6.0)/charset_normalizer (3.5.1) doesn't match a supported version!
warnings.warn(
2026/09/03 09:25:02 INFO: Started server process [46]
2026/09/03 09:25:02 INFO: Waiting for application startup.
2026/09/03 09:25:02 INFO: Application startup complete.
2026/09/03 09:25:04 INFO mlflow.server.jobs.utils: Registered online_scoring_scheduler periodic task (runs every 1 minute)
2026/09/03 09:25:04 INFO mlflow.server.jobs.utils: Registered trace_archival_scheduler periodic task (polls every 1 minute and no-ops when trace archival is disabled or unconfigured)
2026/09/03 09:28:16 INFO: 10.42.16.11:45338 - "GET / HTTP/1.1" 500 Internal Server Error
2026/09/03 09:28:16 ERROR: Exception in ASGI application
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/mlflow/server/fastapi_app.py", line 78, in __call__
await anyio.to_thread.run_sync(self.wsgi, environ, self.start_response)
File "/usr/local/lib/python3.10/site-packages/anyio/to_thread.py", line 65, in run_sync
return await get_async_backend().run_sync_in_worker_thread(
File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 2706, in run_sync_in_worker_thread
return await future
File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 1100, in run
result = context.run(func, *args)
File "/usr/local/lib/python3.10/site-packages/starlette/middleware/wsgi.py", line 150, in wsgi
for chunk in self.app(environ, start_response):
File "/usr/local/lib/python3.10/site-packages/flask/app.py", line 1536, in __call__
return self.wsgi_app(environ, start_response)
File "/usr/local/lib/python3.10/site-packages/flask/app.py", line 1518, in wsgi_app
return response(environ, start_response)
File "/usr/local/lib/python3.10/site-packages/werkzeug/wrappers/response.py", line 577, in __call__
start_response(status, headers)
File "/usr/local/lib/python3.10/site-packages/starlette/middleware/wsgi.py", line 136, in start_response
anyio.from_thread.run(
File "/usr/local/lib/python3.10/site-packages/anyio/_lazyimport.py", line 41, in __getattr__
raise AttributeError(
AttributeError: module 'anyio' has no attribute 'from_thread'
The above exception was the direct cause of the following exception:
+ Exception Group Traceback (most recent call last):
| File "/usr/local/lib/python3.10/site-packages/uvicorn/protocols/http/httptools_impl.py", line 409, in run_asgi
| result = await app( # type: ignore[func-returns-value]
| File "/usr/local/lib/python3.10/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in __call__
| return await self.app(scope, receive, send)
| File "/usr/local/lib/python3.10/site-packages/fastapi/applications.py", line 1160, in __call__
| await super().__call__(scope, receive, send)
| File "/usr/local/lib/python3.10/site-packages/starlette/applications.py", line 90, in __call__
| await self.middleware_stack(scope, receive, send)
| File "/usr/local/lib/python3.10/site-packages/starlette/middleware/errors.py", line 186, in __call__
| raise exc
| File "/usr/local/lib/python3.10/site-packages/starlette/middleware/errors.py", line 164, in __call__
| await self.app(scope, receive, _send)
| File "/usr/local/lib/python3.10/site-packages/starlette/middleware/base.py", line 193, in __call__
| response = await self.dispatch_func(request, call_next)
| File "/usr/local/lib/python3.10/site-packages/mlflow/server/fastapi_app.py", line 138, in gateway_timing_middleware
| return await call_next(request)
| File "/usr/local/lib/python3.10/site-packages/starlette/middleware/base.py", line 168, in call_next
| raise app_exc from app_exc.__cause__ or app_exc.__context__
| File "/usr/local/lib/python3.10/site-packages/starlette/middleware/base.py", line 144, in coro
| await self.app(scope, receive_or_disconnect, send_no_error)
| File "/usr/local/lib/python3.10/site-packages/starlette/middleware/base.py", line 193, in __call__
| response = await self.dispatch_func(request, call_next)
| File "/usr/local/lib/python3.10/site-packages/mlflow/server/fastapi_app.py", line 123, in workspace_context_middleware
| response = await call_next(request)
| File "/usr/local/lib/python3.10/site-packages/starlette/middleware/base.py", line 168, in call_next
| raise app_exc from app_exc.__cause__ or app_exc.__context__
| File "/usr/local/lib/python3.10/site-packages/starlette/middleware/base.py", line 144, in coro
| await self.app(scope, receive_or_disconnect, send_no_error)
| File "/usr/local/lib/python3.10/site-packages/mlflow/server/fastapi_security.py", line 59, in __call__
| return await self.app(scope, receive, send)
| File "/usr/local/lib/python3.10/site-packages/starlette/middleware/cors.py", line 88, in __call__
| await self.app(scope, receive, send)
| File "/usr/local/lib/python3.10/site-packages/mlflow/server/fastapi_security.py", line 114, in __call__
| return await self.app(scope, receive, send)
| File "/usr/local/lib/python3.10/site-packages/mlflow/server/fastapi_security.py", line 99, in __call__
| await self.app(scope, receive, send_wrapper)
| File "/usr/local/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 63, in __call__
| await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
| File "/usr/local/lib/python3.10/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
| raise exc
| File "/usr/local/lib/python3.10/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
| await app(scope, receive, sender)
| File "/usr/local/lib/python3.10/site-packages/fastapi/middleware/asyncexitstack.py", line 18, in __call__
| await self.app(scope, receive, send)
| File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 660, in __call__
| await self.middleware_stack(scope, receive, send)
| File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 680, in app
| await route.handle(scope, receive, send)
| File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 448, in handle
| await self.app(scope, receive, send)
| File "/usr/local/lib/python3.10/site-packages/mlflow/server/fastapi_app.py", line 101, in __call__
| await responder(receive, send)
| File "/usr/local/lib/python3.10/site-packages/mlflow/server/fastapi_app.py", line 75, in __call__
| async with anyio.create_task_group() as task_group:
| File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 847, in __aexit__
| raise BaseExceptionGroup(
| exceptiongroup.ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
+-+---------------- 1 ----------------
| Traceback (most recent call last):
| File "/usr/local/lib/python3.10/site-packages/mlflow/server/fastapi_app.py", line 78, in __call__
| await anyio.to_thread.run_sync(self.wsgi, environ, self.start_response)
| File "/usr/local/lib/python3.10/site-packages/anyio/to_thread.py", line 65, in run_sync
| return await get_async_backend().run_sync_in_worker_thread(
| File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 2706, in run_sync_in_worker_thread
| return await future
| File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 1100, in run
| result = context.run(func, *args)
| File "/usr/local/lib/python3.10/site-packages/starlette/middleware/wsgi.py", line 150, in wsgi
| for chunk in self.app(environ, start_response):
| File "/usr/local/lib/python3.10/site-packages/flask/app.py", line 1536, in __call__
| return self.wsgi_app(environ, start_response)
| File "/usr/local/lib/python3.10/site-packages/flask/app.py", line 1518, in wsgi_app
| return response(environ, start_response)
| File "/usr/local/lib/python3.10/site-packages/werkzeug/wrappers/response.py", line 577, in __call__
| start_response(status, headers)
| File "/usr/local/lib/python3.10/site-packages/starlette/middleware/wsgi.py", line 136, in start_response
| anyio.from_thread.run(
| File "/usr/local/lib/python3.10/site-packages/anyio/_lazyimport.py", line 41, in __getattr__
| raise AttributeError(
| AttributeError: module 'anyio' has no attribute 'from_thread'
+------------------------------------
2026/09/03 09:28:26 WARNING mlflow.server.fastapi_security: Rejected request with invalid Host header: 1-mlflow_server:5006
2026/09/03 09:28:26 INFO: 10.42.16.11:55710 - "GET / HTTP/1.1" 403 ForbiddenThe bug hadn't come up yesterday. |
|
Here is the culprit. Starlette improperly uses the |
|
I have a PR to address this here. Perhaps this thread should've been created as an issue too? I can convert it into one if needed. |
Uh oh!
There was an error while loading. Please reload this page.
After upgrading anyio to version 4.15.0, the application started failing with the following error:
AttributeError: module 'anyio' has no attribute 'abc'
The issue appears to be related to an incompatibility with one of the dependencies that uses anyio.abc. The issue was not observed with the previous AnyIO version.
The current environment includes packages such as elastic-search,open-ai, which has an AnyIO dependency.
All reactions