Running FastAPI app and getting a Starlette error

[bunny-therapist]

I am running a FastAPI app using nginx unit, and using a python script to send multiple sequential (wait for response before sending next) requests to it with the "requests" library. When I reuse the same requests.Session object for all the requests, so that requests.Session.close is not called, then I get intermittent crashes in the application and thus status 500 back.

Traceback (most recent call last):
  File "<path>/venv310/lib/python3.10/site-packages/fastapi/applications.py", line 1054, in __call__
    await super().__call__(scope, receive, send)
  File "<path>/venv310/lib/python3.10/site-packages/starlette/applications.py", line 123, in __call__
    await self.middleware_stack(scope, receive, send)
  File "<path>/venv310/lib/python3.10/site-packages/starlette/middleware/errors.py", line 186, in __call__
    raise exc
  File "<path>/venv310/lib/python3.10/site-packages/starlette/middleware/errors.py", line 164, in __call__
    await self.app(scope, receive, _send)
  File "<path>/venv310/lib/python3.10/site-packages/starlette/middleware/base.py", line 189, in __call__
    with collapse_excgroups():
  File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
  File "<path>/venv310/lib/python3.10/site-packages/starlette/_utils.py", line 91, in collapse_excgroups
    raise exc
  File "<path>/venv310/lib/python3.10/site-packages/starlette/responses.py", line 257, in wrap
    await func()
  File "<path>/venv310/lib/python3.10/site-packages/starlette/middleware/base.py", line 217, in stream_response
    return await super().stream_response(send)
  File "<path>/venv310/lib/python3.10/site-packages/starlette/responses.py", line 246, in stream_response
    async for chunk in self.body_iterator:
  File "<path>/venv310/lib/python3.10/site-packages/starlette/middleware/base.py", line 181, in body_stream
    raise app_exc
  File "<path>/venv310/lib/python3.10/site-packages/starlette/middleware/base.py", line 151, in coro
    await self.app(scope, receive_or_disconnect, send_no_error)
  File "<path>/venv310/lib/python3.10/site-packages/starlette/middleware/base.py", line 189, in __call__
    with collapse_excgroups():
  File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
  File "<path>/venv310/lib/python3.10/site-packages/starlette/_utils.py", line 91, in collapse_excgroups
    raise exc
  File "<path>/venv310/lib/python3.10/site-packages/starlette/middleware/base.py", line 128, in receive_or_disconnect
    message = await wrap(wrapped_receive)
  File "<path>/venv310/lib/python3.10/site-packages/starlette/middleware/base.py", line 123, in wrap
    result = await func()
  File "<path>/venv310/lib/python3.10/site-packages/starlette/middleware/base.py", line 56, in wrapped_receive
    raise RuntimeError(f"Unexpected message received: {msg['type']}")
RuntimeError: Unexpected message received: http.request

The error is coming from this line: https://github.com/encode/starlette/blob/master/starlette/middleware/base.py#L56
which appears to be starlette waiting for a client disconnect but instead getting the next request. (No idea why starlette wants the request to be closed instead of allowing it to be reused for the next request, but maybe there is something I am not aware of.)

I would (and will) ask about this in the starlette github, but I am posting here because I did not encounter this issue when running my FastAPI app with uvicorn, it only happens with nginx unit, so at a minimum, it appears that something is different here which is causing problems, and this is probably something that should be identified.

I was really happy when I discovered nginx unit; this issue is currently the only thing that is preventing me from adopting it.

[ac000]

Do you have a minimal reproducer?

[bunny-therapist]

I am working on it, but it is incredibly unstable.

imho, a minimal example should just be a FastAPI app with some middleware; possibly requiring a body in either request or response.

I will keep trying tomorrow. One question though:

Am I expected to restart the "unit" service after updating the configuration using the "/config" endpoint? In the log, the application seems to restart even with doing that, but I keep running into unrecoverable problems (e.g. the app ending up in an "invalid state") and then have to restart unit.

[bunny-therapist]

Minimal reproducer:

FastAPI app ('app.py'):

from fastapi import FastAPI
from fastapi.responses import JSONResponse
from pydantic import BaseModel

class RequestBody(BaseModel):
    number: int

app = FastAPI()


@app.middleware("http")
async def my_middleware(request, call_next):
    response = await call_next(request)
    return response


@app.post("/predict")
async def post_predict(body: RequestBody):
    return JSONResponse(content={"got": body.number})

Client script:

import requests

N = 2
with requests.Session() as session:
    for _ in range(N):
        r = session.post("http://127.0.0.1:8000/predict", json={"number": 1})
        r.raise_for_status()

When I run the client script the first time I get status 503 back and I can see the error message in the unit log. When I rerun the script, I still see the same error message in the unit log, but the client gets an ok response back.

The unit config is very bare-bones:

{
    "listeners": {
        "127.0.0.1:8000": {
            "pass": "applications/fastapi"
        }
    },

    "applications": {
        "fastapi": {
            "type": "python 3.10",
             "path": "<path",
            "home": "<path>/venv310/",
            "user": "<me>",
            "group": "<me>",
            "module": "app",
            "callable": "app",
	    "protocol": "asgi"
        }
    }
}

[ac000]

I am working on it, but it is incredibly unstable.

imho, a minimal example should just be a FastAPI app with some middleware; possibly requiring a body in either request or response.

Heh, assume I know nothing about FastAPI apps...

I will keep trying tomorrow. One question though:

Thanks!

Am I expected to restart the "unit" service after updating the configuration using the "/config" endpoint? In the log, the application seems to restart even with doing that, but I keep running into unrecoverable problems (e.g. the app ending up in an "invalid state") and then have to restart unit.

You shouldn't need to restart it generally... but there are times you do...

[bunny-therapist]

If I just have a "get" endpoint with no request nor response body, I get this error every time:

Exception in callback Future.set_result({'type': 'http.disconnect'})
handle: <Handle Future.set_result({'type': 'http.disconnect'})>
Traceback (most recent call last):
  File "/usr/lib/python3.10/asyncio/events.py", line 80, in _run
    self._context.run(self._callback, *self._args)
asyncio.exceptions.InvalidStateError: invalid state
Exception in callback Future.set_result({'type': 'http.disconnect'})
handle: <Handle Future.set_result({'type': 'http.disconnect'})>
Traceback (most recent call last):
  File "/usr/lib/python3.10/asyncio/events.py", line 80, in _run
    self._context.run(self._callback, *self._args)
asyncio.exceptions.InvalidStateError: invalid state

If I remove the middleware, the error disappears.

To me, it seems like there is some problem with nginx unit and fastapi/starlette middleware.

[bunny-therapist]

The GET requests pass, I get back OK. However, the "InvalidStateError" shows up in the unit log every single time.

[bunny-therapist]

Here is how to add a simple GET endpoint. Just put that in the app.py I posted above. As long as the middleware is in the app, you get the error in the log when calling this endpoint.

from fastapi.responses import  Response

@app.get("/data")
async def get_data():
    return Response()

[bunny-therapist]

The simple GET error (invalid state) appears to go away when I go to fastapi==0.107.0 starlette==0.28.0
The POST error (unexpected message) is still there. I will try to go even older.

[bunny-therapist]

With fastapi==0.106.0 starlette==0.27.0 the POST error also goes away, probably because starlette 0.27.0 does not have this commit, which added the code where the error is coming from: encode/starlette@554b9e2

[ac000]

Thanks, I'll have a look at your reproducer.

What is the middleware you mention?

[bunny-therapist]

This part:

@app.middleware("http")
async def my_middleware(request, call_next):
    response = await call_next(request)
    return response

[bunny-therapist]

FastAPI middleware is based on Starlette middleware (because FastAPI is based on Starlette). The errors seem to originate from Starlette's middleware code, and the errors disappear if I remove the middleware from the app.

[ac000]

OK, I've reproduced this.

This is perhaps better tracked in an issue. I'll transfer this over.