Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RuntimeError in uvicorn for some requests #111

Closed
ChillarAnand opened this issue Jul 4, 2018 · 35 comments
Closed

RuntimeError in uvicorn for some requests #111

ChillarAnand opened this issue Jul 4, 2018 · 35 comments

Comments

@ChillarAnand
Copy link
Contributor

uvicorn==0.2.5 is throwing errors for some requests.

ERROR: Exception in ASGI application
Traceback (most recent call last):
  File "/home/chillar/.virtualenvs/library/lib/python3.6/site-packages/uvicorn/protocols/http/httptools.py", line 196, in run_asgi
    result = await asgi(self.receive, self.send)
  File "/home/chillar/.virtualenvs/library/lib/python3.6/site-packages/channels/http.py", line 190, in __call__
    await self.handle(body)
  File "/home/chillar/.virtualenvs/library/lib/python3.6/site-packages/asgiref/sync.py", line 110, in __call__
    return await asyncio.wait_for(future, timeout=None)
  File "/usr/lib/python3.6/asyncio/tasks.py", line 339, in wait_for
    return (yield from fut)
  File "/usr/lib/python3.6/concurrent/futures/thread.py", line 56, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/home/chillar/.virtualenvs/library/lib/python3.6/site-packages/asgiref/sync.py", line 125, in thread_handler
    return self.func(*args, **kwargs)
  File "/home/chillar/.virtualenvs/library/lib/python3.6/site-packages/channels/http.py", line 229, in handle
    self.send(response_message)
  File "/home/chillar/.virtualenvs/library/lib/python3.6/site-packages/asgiref/sync.py", line 64, in __call__
    return call_result.result()
  File "/usr/lib/python3.6/concurrent/futures/_base.py", line 432, in result
    return self.__get_result()
  File "/usr/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
  File "/home/chillar/.virtualenvs/library/lib/python3.6/site-packages/asgiref/sync.py", line 78, in main_wrap
    result = await self.awaitable(*args, **kwargs)
  File "/home/chillar/.virtualenvs/library/lib/python3.6/site-packages/uvicorn/protocols/http/httptools.py", line 308, in send
    protocol.transport.write(body)
  File "uvloop/handles/stream.pyx", line 636, in uvloop.loop.UVStream.write
  File "uvloop/handles/handle.pyx", line 165, in uvloop.loop.UVHandle._ensure_alive
RuntimeError: unable to perform operation on <TCPTransport closed=True reading=False 0x1a48ad8>; the handler is closed
@ChillarAnand ChillarAnand changed the title RuntimeError in uvloop RuntimeError in uvicorn for some requests Jul 4, 2018
@tomchristie
Copy link
Member

Looks like it's a case when sending a response and the connection has closed prematurely.

Might be resolved by if self.disconnected: return at the start of the send() method.

Is it easy to reproduce?

@ChillarAnand
Copy link
Contributor Author

I am not sure how to reproduce. In one project it is happening for every 5 requests or so and it is failing randomly. In other projects its not happening at all.

@tomchristie
Copy link
Member

Okay. I’m guessing you’re not seeing error on the client side? Looks like we should issue a warning, and then return in this case. The application is sending data but the connection has been closed by the client, which isn’t really an error case, but is worth logging with a warning. Will see if can figure what Daphne does in this cases.

@tomchristie
Copy link
Member

tomchristie commented Jul 4, 2018

I think daphne just silently ignores this case. Do you see the same behaviour running with ‘—http h11’?

@ChillarAnand
Copy link
Contributor Author

Okay. I’m guessing you’re not seeing error on the client side?

No

For h11 also, it is throwing error

ERROR: Exception in callback UVTransport._call_connection_lost
handle: <Handle UVTransport._call_connection_lost>
Traceback (most recent call last):
  File "/home/chillar/.virtualenvs/library/lib/python3.6/site-packages/h11/_state.py", line 249, in _fire_event_triggered_transitions
    new_state = EVENT_TRIGGERED_TRANSITIONS[role][state][event_type]
KeyError: <class 'h11._events.ConnectionClosed'>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "uvloop/cbhandles.pyx", line 70, in uvloop.loop.Handle._run
  File "uvloop/handles/basetransport.pyx", line 202, in uvloop.loop.UVBaseTransport._call_connection_lost
  File "/home/chillar/.virtualenvs/library/lib/python3.6/site-packages/uvicorn/protocols/http/h11.py", line 79, in connection_lost
    self.conn.send(event)
  File "/home/chillar/.virtualenvs/library/lib/python3.6/site-packages/h11/_connection.py", line 464, in send
    data_list = self.send_with_data_passthrough(event)
  File "/home/chillar/.virtualenvs/library/lib/python3.6/site-packages/h11/_connection.py", line 490, in send_with_data_passthrough
    self._process_event(self.our_role, event)
  File "/home/chillar/.virtualenvs/library/lib/python3.6/site-packages/h11/_connection.py", line 238, in _process_event
    self._cstate.process_event(role, type(event), server_switch_event)
  File "/home/chillar/.virtualenvs/library/lib/python3.6/site-packages/h11/_state.py", line 238, in process_event
    self._fire_event_triggered_transitions(role, event_type)
  File "/home/chillar/.virtualenvs/library/lib/python3.6/site-packages/h11/_state.py", line 253, in _fire_event_triggered_transitions
    .format(event_type, role, self.states[role]))
h11._util.LocalProtocolError: can't handle event type <class 'h11._events.ConnectionClosed'> for SERVER in state SEND_BODY

ERROR: Exception in ASGI application
Traceback (most recent call last):
  File "/home/chillar/.virtualenvs/library/lib/python3.6/site-packages/uvicorn/protocols/http/h11.py", line 171, in run_asgi
    result = await asgi(self.receive, self.send)
  File "/home/chillar/.virtualenvs/library/lib/python3.6/site-packages/channels/http.py", line 190, in __call__
    await self.handle(body)
  File "/home/chillar/.virtualenvs/library/lib/python3.6/site-packages/asgiref/sync.py", line 110, in __call__
    return await asyncio.wait_for(future, timeout=None)
  File "/usr/lib/python3.6/asyncio/tasks.py", line 339, in wait_for
    return (yield from fut)
  File "/usr/lib/python3.6/concurrent/futures/thread.py", line 56, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/home/chillar/.virtualenvs/library/lib/python3.6/site-packages/asgiref/sync.py", line 125, in thread_handler
    return self.func(*args, **kwargs)
  File "/home/chillar/.virtualenvs/library/lib/python3.6/site-packages/channels/http.py", line 229, in handle
    self.send(response_message)
  File "/home/chillar/.virtualenvs/library/lib/python3.6/site-packages/asgiref/sync.py", line 64, in __call__
    return call_result.result()
  File "/usr/lib/python3.6/concurrent/futures/_base.py", line 432, in result
    return self.__get_result()
  File "/usr/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
  File "/home/chillar/.virtualenvs/library/lib/python3.6/site-packages/asgiref/sync.py", line 78, in main_wrap
    result = await self.awaitable(*args, **kwargs)
  File "/home/chillar/.virtualenvs/library/lib/python3.6/site-packages/uvicorn/protocols/http/h11.py", line 261, in send
    output = protocol.conn.send(event)
  File "/home/chillar/.virtualenvs/library/lib/python3.6/site-packages/h11/_connection.py", line 464, in send
    data_list = self.send_with_data_passthrough(event)
  File "/home/chillar/.virtualenvs/library/lib/python3.6/site-packages/h11/_connection.py", line 480, in send_with_data_passthrough
    "Can't send data when our state is ERROR")
h11._util.LocalProtocolError: Can't send data when our state is ERROR

@tomchristie
Copy link
Member

Released as 0.2.6

@oTree-org
Copy link

FYI I am getting this error with uvicorn 0.11.5 and h11 0.9.0, on a Django channels project. I'm running uvicorn with just the basic host/port options. The error happens only sometimes and has not had any impact on my app's functionality.

ERROR:    Exception in callback H11Protocol.timeout_keep_alive_handler()
handle: <TimerHandle when=112443.109 H11Protocol.timeout_keep_alive_handler()>
Traceback (most recent call last):
  File "c:\otree\ve38\lib\site-packages\h11\_state.py", line 249, in _fire_event_triggered_transitions
    new_state = EVENT_TRIGGERED_TRANSITIONS[role][state][event_type]
KeyError: <class 'h11._events.ConnectionClosed'>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\wicke\AppData\Local\Programs\Python\Python38\lib\asyncio\events.py", line 81, in _run
    self._context.run(self._callback, *self._args)
  File "c:\otree\ve38\lib\site-packages\uvicorn\protocols\http\h11_impl.py", line 339, in timeout_keep_alive_handler
    self.conn.send(event)
  File "c:\otree\ve38\lib\site-packages\h11\_connection.py", line 469, in send
    data_list = self.send_with_data_passthrough(event)
  File "c:\otree\ve38\lib\site-packages\h11\_connection.py", line 494, in send_with_data_passthrough
    self._process_event(self.our_role, event)
  File "c:\otree\ve38\lib\site-packages\h11\_connection.py", line 242, in _process_event
    self._cstate.process_event(role, type(event), server_switch_event)
  File "c:\otree\ve38\lib\site-packages\h11\_state.py", line 238, in process_event
    self._fire_event_triggered_transitions(role, event_type)
  File "c:\otree\ve38\lib\site-packages\h11\_state.py", line 251, in _fire_event_triggered_transitions
    raise LocalProtocolError(
h11._util.LocalProtocolError: can't handle event type ConnectionClosed when role=SERVER and state=SEND_RESPONSE
ERROR:    Exception in callback H11Protocol.timeout_keep_alive_handler()
handle: <TimerHandle when=112443.109 H11Protocol.timeout_keep_alive_handler()>
Traceback (most recent call last):
  File "c:\otree\ve38\lib\site-packages\h11\_state.py", line 249, in _fire_event_triggered_transitions
    new_state = EVENT_TRIGGERED_TRANSITIONS[role][state][event_type]
KeyError: <class 'h11._events.ConnectionClosed'>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\wicke\AppData\Local\Programs\Python\Python38\lib\asyncio\events.py", line 81, in _run
    self._context.run(self._callback, *self._args)
  File "c:\otree\ve38\lib\site-packages\uvicorn\protocols\http\h11_impl.py", line 339, in timeout_keep_alive_handler
    self.conn.send(event)
  File "c:\otree\ve38\lib\site-packages\h11\_connection.py", line 469, in send
    data_list = self.send_with_data_passthrough(event)
  File "c:\otree\ve38\lib\site-packages\h11\_connection.py", line 494, in send_with_data_passthrough
    self._process_event(self.our_role, event)
  File "c:\otree\ve38\lib\site-packages\h11\_connection.py", line 242, in _process_event
    self._cstate.process_event(role, type(event), server_switch_event)
  File "c:\otree\ve38\lib\site-packages\h11\_state.py", line 238, in process_event
    self._fire_event_triggered_transitions(role, event_type)
  File "c:\otree\ve38\lib\site-packages\h11\_state.py", line 251, in _fire_event_triggered_transitions
    raise LocalProtocolError(
h11._util.LocalProtocolError: can't handle event type ConnectionClosed when role=SERVER and state=SEND_RESPONSE
INFO:     127.0.0.1:52787 - "GET /p/oypfo5xq/misc_3p/PickWinner/2/ HTTP/1.1" 302 Found
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "c:\otree\ve38\lib\site-packages\uvicorn\protocols\http\h11_impl.py", line 384, in run_asgi
    result = await app(self.scope, self.receive, self.send)
  File "c:\otree\ve38\lib\site-packages\uvicorn\middleware\proxy_headers.py", line 45, in __call__
    return await self.app(scope, receive, send)
  File "c:\otree\ve38\lib\site-packages\uvicorn\middleware\asgi2.py", line 7, in __call__
    await instance(receive, send)
  File "c:\otree\ve38\lib\site-packages\channels\http.py", line 192, in __call__
    await self.handle(body_stream)
  File "c:\otree\ve38\lib\site-packages\asgiref\sync.py", line 269, in __call__
    ret = await asyncio.wait_for(future, timeout=None)
  File "C:\Users\wicke\AppData\Local\Programs\Python\Python38\lib\asyncio\tasks.py", line 455, in wait_for
    return await fut
  File "C:\Users\wicke\AppData\Local\Programs\Python\Python38\lib\concurrent\futures\thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "c:\otree\ve38\lib\site-packages\asgiref\sync.py", line 314, in thread_handler
    return func(*args, **kwargs)
  File "c:\otree\ve38\lib\site-packages\channels\http.py", line 250, in handle
    self.send(response_message)
  File "c:\otree\ve38\lib\site-packages\asgiref\sync.py", line 120, in __call__
    return call_result.result()
  File "C:\Users\wicke\AppData\Local\Programs\Python\Python38\lib\concurrent\futures\_base.py", line 432, in result
    return self.__get_result()
  File "C:\Users\wicke\AppData\Local\Programs\Python\Python38\lib\concurrent\futures\_base.py", line 388, in __get_result
    raise self._exception
  File "c:\otree\ve38\lib\site-packages\asgiref\sync.py", line 180, in main_wrap
    result = await self.awaitable(*args, **kwargs)
  File "c:\otree\ve38\lib\site-packages\uvicorn\protocols\http\h11_impl.py", line 461, in send
    output = self.conn.send(event)
  File "c:\otree\ve38\lib\site-packages\h11\_connection.py", line 469, in send
    data_list = self.send_with_data_passthrough(event)
  File "c:\otree\ve38\lib\site-packages\h11\_connection.py", line 484, in send_with_data_passthrough
    raise LocalProtocolError("Can't send data when our state is ERROR")
h11._util.LocalProtocolError: Can't send data when our state is ERROR

Happy to provide extra info.

@arxell
Copy link

arxell commented Oct 1, 2020

the same problem
uvicorn==0.12.1
h11==0.9.0
python==3.8.5

2020-10-01 08:21:53 - asyncio - ERROR - Exception in callback H11Protocol.timeout_keep_alive_handler()
handle: <TimerHandle when=852691.425380342 H11Protocol.timeout_keep_alive_handler()>
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/h11/_state.py", line 249, in _fire_event_triggered_transitions
    new_state = EVENT_TRIGGERED_TRANSITIONS[role][state][event_type]
KeyError: <class 'h11._events.ConnectionClosed'>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/asyncio/events.py", line 81, in _run
    self._context.run(self._callback, *self._args)
  File "/usr/local/lib/python3.8/site-packages/uvicorn/protocols/http/h11_impl.py", line 344, in timeout_keep_alive_handler
    self.conn.send(event)
  File "/usr/local/lib/python3.8/site-packages/h11/_connection.py", line 469, in send
    data_list = self.send_with_data_passthrough(event)
  File "/usr/local/lib/python3.8/site-packages/h11/_connection.py", line 494, in send_with_data_passthrough
    self._process_event(self.our_role, event)
  File "/usr/local/lib/python3.8/site-packages/h11/_connection.py", line 242, in _process_event
    self._cstate.process_event(role, type(event), server_switch_event)
  File "/usr/local/lib/python3.8/site-packages/h11/_state.py", line 238, in process_event
    self._fire_event_triggered_transitions(role, event_type)
  File "/usr/local/lib/python3.8/site-packages/h11/_state.py", line 251, in _fire_event_triggered_transitions
    raise LocalProtocolError(
h11._util.LocalProtocolError: can't handle event type ConnectionClosed when role=SERVER and state=SEND_RESPONSE

@euri10 euri10 reopened this Oct 1, 2020
@euri10
Copy link
Member

euri10 commented Oct 1, 2020

could you do 2 things please to try nail that down,

  1. try with h11 > 0.9, seens like you both @oTree-org and @arxell have the same issue on that same version
  2. give us a minimal reproductible example to play with, please use the issue template that asks for a lot of information but for a good purpose

thanks

@arxell
Copy link

arxell commented Oct 1, 2020

I cant update h11 to 0.10.0 because latest version httpcore (0.11.1):

    install_requires=["h11>=0.8,<0.10", "sniffio==1.*"],

@oTree-org
Copy link

FYI i updated to h11==0.11.0 and still getting the error. (Later I will work on providing the full bug report as requested, thank you.)

@oTree-org
Copy link

I am working on creating a minimal reproducible example, but I am proceeding blindly since I don't really understand the error message, so I am just randomly adding / removing stuff from my code and trying different ways of loading the page.

Could someone explain the error message a bit? That would help me craft a better MWE.

I can get it to reproduce (but only sporadically) by quickly reloading a simple Django view that just loads this HTML:

<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="UTF-8">
  <title>Title</title>
</head>
<body>

</body>
</html>

I'm getting these errors:

ERROR Exception in callback H11Protocol.timeout_keep_alive_handler()
handle: <TimerHandle when=196231.328 H11Protocol.timeout_keep_alive_handler()>
Traceback (most recent call last):
  File "c:\otree\ve-uvicorn\lib\site-packages\h11\_state.py", line 249, in _fire_event_triggered_transitions
    new_state = EVENT_TRIGGERED_TRANSITIONS[role][state][event_type]
KeyError: <class 'h11._events.ConnectionClosed'>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\wicke\AppData\Local\Programs\Python\Python38\lib\asyncio\events.py", line 81, in _run
    self._context.run(self._callback, *self._args)
  File "c:\otree\ve-uvicorn\lib\site-packages\uvicorn\protocols\http\h11_impl.py", line 344, in timeout_keep_alive_handler
    self.conn.send(event)
  File "c:\otree\ve-uvicorn\lib\site-packages\h11\_connection.py", line 469, in send
    data_list = self.send_with_data_passthrough(event)
  File "c:\otree\ve-uvicorn\lib\site-packages\h11\_connection.py", line 494, in send_with_data_passthrough
    self._process_event(self.our_role, event)
  File "c:\otree\ve-uvicorn\lib\site-packages\h11\_connection.py", line 242, in _process_event
    self._cstate.process_event(role, type(event), server_switch_event)
  File "c:\otree\ve-uvicorn\lib\site-packages\h11\_state.py", line 238, in process_event
    self._fire_event_triggered_transitions(role, event_type)
  File "c:\otree\ve-uvicorn\lib\site-packages\h11\_state.py", line 251, in _fire_event_triggered_transitions
    raise LocalProtocolError(
h11._util.LocalProtocolError: can't handle event type ConnectionClosed when role=SERVER and state=SEND_RESPONSE
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "c:\otree\ve-uvicorn\lib\site-packages\uvicorn\protocols\http\h11_impl.py", line 389, in run_asgi
    result = await app(self.scope, self.receive, self.send)
  File "c:\otree\ve-uvicorn\lib\site-packages\uvicorn\middleware\proxy_headers.py", line 45, in __call__
    return await self.app(scope, receive, send)
  File "c:\otree\ve-uvicorn\lib\site-packages\uvicorn\middleware\asgi2.py", line 7, in __call__
    await instance(receive, send)
  File "c:\otree\core\channels\http.py", line 192, in __call__
    await self.handle(body_stream)
  File "c:\otree\ve-uvicorn\lib\site-packages\asgiref\sync.py", line 269, in __call__
    ret = await asyncio.wait_for(future, timeout=None)
  File "C:\Users\wicke\AppData\Local\Programs\Python\Python38\lib\asyncio\tasks.py", line 455, in wait_for
    return await fut
  File "C:\Users\wicke\AppData\Local\Programs\Python\Python38\lib\concurrent\futures\thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "c:\otree\ve-uvicorn\lib\site-packages\asgiref\sync.py", line 314, in thread_handler
    return func(*args, **kwargs)
  File "c:\otree\core\channels\http.py", line 250, in handle
    self.send(response_message)
  File "c:\otree\ve-uvicorn\lib\site-packages\asgiref\sync.py", line 120, in __call__
    return call_result.result()
  File "C:\Users\wicke\AppData\Local\Programs\Python\Python38\lib\concurrent\futures\_base.py", line 432, in result
    return self.__get_result()
  File "C:\Users\wicke\AppData\Local\Programs\Python\Python38\lib\concurrent\futures\_base.py", line 388, in __get_result
    raise self._exception
  File "c:\otree\ve-uvicorn\lib\site-packages\asgiref\sync.py", line 180, in main_wrap
    result = await self.awaitable(*args, **kwargs)
  File "c:\otree\ve-uvicorn\lib\site-packages\uvicorn\protocols\http\h11_impl.py", line 466, in send
    output = self.conn.send(event)
  File "c:\otree\ve-uvicorn\lib\site-packages\h11\_connection.py", line 469, in send
    data_list = self.send_with_data_passthrough(event)
  File "c:\otree\ve-uvicorn\lib\site-packages\h11\_connection.py", line 484, in send_with_data_passthrough
    raise LocalProtocolError("Can't send data when our state is ERROR")
h11._util.LocalProtocolError: Can't send data when our state is ERROR

@oTree-org
Copy link

(note, this error is not the same as the original one in this bug report, so let me know if i need to open a new issue)

@euri10
Copy link
Member

euri10 commented Nov 18, 2020

reading the message it seems like the keep-alive timer is at stake, we recently fixed 2 races conditions in #848 and #832 and the last one seems directly related, can you reproduce with the latest master ?
now to be fair @oTree-org I dont get the message either :) plus I'm clueless on Django, this said if you send me a gist with the django app I'll gladly spam my F5 button tomorrow and have a look

@oTree-org
Copy link

@euri10 the error is gone in the latest master, thank you! 👍

(I am still getting this issue: #757 but will continue the discussion over there.)

@euri10
Copy link
Member

euri10 commented Nov 19, 2020

Fixed by #832 it seems !😎

@euri10 euri10 closed this as completed Nov 19, 2020
@asvatov
Copy link

asvatov commented May 24, 2022

Still getting similar exceptions (can't provide some context, just catch it occasionally in my logs)
My stack:

  • uvicorn==0.15.0
  • starlette==0.14.2
  • h11==0.13.0
  • python==3.9.4
[ERROR] Exception in ASGI application
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/uvicorn/protocols/http/h11_impl.py", line 373, in run_asgi
     result = await app(self.scope, self.receive, self.send)
   File "/usr/local/lib/python3.9/site-packages/uvicorn/middleware/proxy_headers.py", line 75, in __call__
     return await self.app(scope, receive, send)
   File "/usr/local/lib/python3.9/site-packages/fastapi/applications.py", line 208, in __call__
     await super().__call__(scope, receive, send)
   File "/usr/local/lib/python3.9/site-packages/starlette/applications.py", line 112, in __call__
     await self.middleware_stack(scope, receive, send)
   File "/usr/local/lib/python3.9/site-packages/starlette/middleware/errors.py", line 181, in __call__
     raise exc from None
   File "/usr/local/lib/python3.9/site-packages/starlette/middleware/errors.py", line 159, in __call__
     await self.app(scope, receive, _send)
   File "/usr/local/lib/python3.9/site-packages/starlette/middleware/cors.py", line 78, in __call__
     await self.app(scope, receive, send)
   File "/usr/local/lib/python3.9/site-packages/starlette/middleware/base.py", line 26, in __call__
     await response(scope, receive, send)
   File "/usr/local/lib/python3.9/site-packages/starlette/responses.py", line 224, in __call__
     await run_until_first_complete(
   File "/usr/local/lib/python3.9/site-packages/starlette/concurrency.py", line 24, in run_until_first_complete
     [task.result() for task in done]
   File "/usr/local/lib/python3.9/site-packages/starlette/concurrency.py", line 24, in <listcomp>
     [task.result() for task in done]
   File "/usr/local/lib/python3.9/site-packages/starlette/responses.py", line 209, in stream_response
     await send(
   File "/usr/local/lib/python3.9/site-packages/starlette/middleware/errors.py", line 156, in _send
     await send(message)
   File "/usr/local/lib/python3.9/site-packages/uvicorn/protocols/http/h11_impl.py", line 452, in send
     output = self.conn.send(event)
   File "/usr/local/lib/python3.9/site-packages/h11/_connection.py", line 510, in send
     data_list = self.send_with_data_passthrough(event)
   File "/usr/local/lib/python3.9/site-packages/h11/_connection.py", line 525, in send_with_data_passthrough
     raise LocalProtocolError("Can't send data when our state is ERROR")
 h11._util.LocalProtocolError: Can't send data when our state is ERROR

@Kludex
Copy link
Sponsor Member

Kludex commented May 24, 2022

Those versions are too old.

@asvatov
Copy link

asvatov commented May 24, 2022

@Kludex Do you mean that upgrade to the latest version of uvicorn can resolve it to me?

@Kludex
Copy link
Sponsor Member

Kludex commented May 24, 2022

I hope so... I cannot say. But the first thing to do is upgrade those.

@danielaptekman
Copy link

Getting the same sporadic exceptions described by @arxell and @oTree-org (Exception in callback H11Protocol.timeout_keep_alive_handler())

Python==3.9.12
h11==0.13.0
uvicorn==0.17.6
starlette==0.19.1

Any leads?

Thanks!

@Tankilevitch
Copy link

Getting the same errors:

h11._util.LocalProtocolError: Can't send data when our state is ERROR\n","stream":"stderr","time":"2022-07-26T13:06:13.058314397Z"}
{"log":"ERROR [uvicorn.error] Exception in ASGI application\n","stream":"stderr","time":"2022-07-26T13:06:13.05831706Z"}
{"log":"Traceback (most recent call last):\n","stream":"stderr","time":"2022-07-26T13:06:13.058319577Z"}
{"log":"  File \"/usr/local/lib/python3.7/site-packages/uvicorn/protocols/http/h11_impl.py\", line 366, in run_asgi\n","stream":"stderr","time":"2022-07-26T13:06:13.058322115Z"}
{"log":"    result = await app(self.scope, self.receive, self.send)\n","stream":"stderr","time":"2022-07-26T13:06:13.05832492Z"}
{"log":"  File \"/usr/local/lib/python3.7/site-packages/uvicorn/middleware/proxy_headers.py\", line 75, in __call__\n","stream":"stderr","time":"2022-07-26T13:06:13.058327361Z"}
{"log":"    return await self.app(scope, receive, send)\n","stream":"stderr","time":"2022-07-26T13:06:13.058330114Z"}
{"log":"  File \"/usr/local/lib/python3.7/site-packages/fastapi/applications.py\", line 269, in __call__\n","stream":"stderr","time":"2022-07-26T13:06:13.058332663Z"}
{"log":"    await super().__call__(scope, receive, send)\n","stream":"stderr","time":"2022-07-26T13:06:13.05833534Z"}
{"log":"  File \"/usr/local/lib/python3.7/site-packages/starlette/applications.py\", line 124, in __call__\n","stream":"stderr","time":"2022-07-26T13:06:13.058337761Z"}
{"log":"    await self.middleware_stack(scope, receive, send)\n","stream":"stderr","time":"2022-07-26T13:06:13.058340552Z"}
{"log":"  File \"/usr/local/lib/python3.7/site-packages/starlette/middleware/errors.py\", line 184, in __call__\n","stream":"stderr","time":"2022-07-26T13:06:13.058342962Z"}
{"log":"    raise exc\n","stream":"stderr","time":"2022-07-26T13:06:13.058347714Z"}
{"log":"  File \"/usr/local/lib/python3.7/site-packages/starlette/middleware/errors.py\", line 162, in __call__\n","stream":"stderr","time":"2022-07-26T13:06:13.058350253Z"}
{"log":"    await self.app(scope, receive, _send)\n","stream":"stderr","time":"2022-07-26T13:06:13.05835295Z"}
{"log":"  File \"/usr/local/lib/python3.7/site-packages/starlette/middleware/base.py\", line 69, in __call__\n","stream":"stderr","time":"2022-07-26T13:06:13.058355367Z"}
{"log":"    await response(scope, receive, send)\n","stream":"stderr","time":"2022-07-26T13:06:13.058358138Z"}
{"log":"  File \"/usr/local/lib/python3.7/site-packages/starlette/responses.py\", line 260, in __call__\n","stream":"stderr","time":"2022-07-26T13:06:13.05836055Z"}
{"log":"    await wrap(partial(self.listen_for_disconnect, receive))\n","stream":"stderr","time":"2022-07-26T13:06:13.058363207Z"}
{"log":"  File \"/usr/local/lib/python3.7/site-packages/anyio/_backends/_asyncio.py\", line 662, in __aexit__\n","stream":"stderr","time":"2022-07-26T13:06:13.058365713Z"}
{"log":"    raise exceptions[0]\n","stream":"stderr","time":"2022-07-26T13:06:13.058368359Z"}
{"log":"  File \"/usr/local/lib/python3.7/site-packages/anyio/_backends/_asyncio.py\", line 702, in _run_wrapped_task\n","stream":"stderr","time":"2022-07-26T13:06:13.058370723Z"}
{"log":"    await coro\n","stream":"stderr","time":"2022-07-26T13:06:13.058373401Z"}
{"log":"  File \"/usr/local/lib/python3.7/site-packages/starlette/responses.py\", line 256, in wrap\n","stream":"stderr","time":"2022-07-26T13:06:13.058375762Z"}
{"log":"    await func()\n","stream":"stderr","time":"2022-07-26T13:06:13.05837841Z"}
{"log":"  File \"/usr/local/lib/python3.7/site-packages/starlette/responses.py\", line 242, in stream_response\n","stream":"stderr","time":"2022-07-26T13:06:13.058380829Z"}
{"log":"    \"headers\": self.raw_headers,\n","stream":"stderr","time":"2022-07-26T13:06:13.058383902Z"}
{"log":"  File \"/usr/local/lib/python3.7/site-packages/starlette/middleware/errors.py\", line 159, in _send\n","stream":"stderr","time":"2022-07-26T13:06:13.058386497Z"}
{"log":"    await send(message)\n","stream":"stderr","time":"2022-07-26T13:06:13.058389186Z"}
{"log":"  File \"/usr/local/lib/python3.7/site-packages/uvicorn/protocols/http/h11_impl.py\", line 445, in send\n","stream":"stderr","time":"2022-07-26T13:06:13.058391526Z"}
{"log":"    output = self.conn.send(event)\n","stream":"stderr","time":"2022-07-26T13:06:13.058394252Z"}
{"log":"  File \"/usr/local/lib/python3.7/site-packages/h11/_connection.py\", line 510, in send\n","stream":"stderr","time":"2022-07-26T13:06:13.058397205Z"}
{"log":"    data_list = self.send_with_data_passthrough(event)\n","stream":"stderr","time":"2022-07-26T13:06:13.058399798Z"}
{"log":"  File \"/usr/local/lib/python3.7/site-packages/h11/_connection.py\", line 525, in send_with_data_passthrough\n","stream":"stderr","time":"2022-07-26T13:06:13.058402358Z"}
{"log":"    raise LocalProtocolError(\"Can't send data when our state is ERROR\")\n","stream":"stderr","time":"2022-07-26T13:06:13.058405043Z"}
{"log":"h11._util.LocalProtocolError: Can't send data when our state is ERROR\n","stream":"stderr","time":"2022-07-26T13:06:13.05840769Z"}
python==3.7.13
h11==0.13.0
uvicorn==0.17.6
starlette==0.19.1

@sk-
Copy link

sk- commented Aug 17, 2022

We are also getting a bunch of these errors.

Our dependencies are:

h11: 0.12.0
uvicorn: 0.18.2
fastapi: 0.79.0
starlette: 0.19.1
sentry-sdk: 1.9.4

@Kludex
Copy link
Sponsor Member

Kludex commented Aug 17, 2022

This issue was resolved years ago, and there's too much information here.

If interested, please open a discussion with a minimal reproducible example.

@r1b
Copy link

r1b commented Jan 2, 2023

For the commenters above - if the following applies to you:

  • You are using FastAPI<0.87.0
  • You have path operations defined using def instead of async def
  • You have defined custom middleware

..then you may be susceptible to encode/starlette#919, which can cause these errors in uvicorn:

  • LocalProtocolError: Can't send data when our state is ERROR
  • LocalProtocolError: can't handle event type ConnectionClosed when role=SERVER and state=SEND_RESPONSE

This issue was resolved in starlette==0.21.0, which was added in FastAPI==0.87.0. Bumping to FastAPI>=0.87.0 should fix your application.

Apologies for responding to an old issue, this page is the top search result for the above errors.

@vdytyniak-exos
Copy link

vdytyniak-exos commented Feb 28, 2023

@r1b I am still getting this error when trying to test def endpoint with ~50 users concurrency. Do you have any idea why?
FastAPI==0.92.0

Update:
After removing middleware I don't see this issue. Does anyone has ideas why it breaks the server?

@ClementWalter
Copy link

I'm using name fastapi==0.94.1 and had continuous LocalProtocolError: Can't send data when our state is ERROR using this middleware

class RequestContextLogMiddleware(BaseHTTPMiddleware):
    async def set_body(self, request: Request):
        receive_ = await request._receive()

        async def receive():
            return receive_

        request._receive = receive

    async def dispatch(self, request: Request, call_next):
        await self.set_body(request)
        body = await request.json()
        logger.debug(f"RPC request:\n\t{body}")
        response = await call_next(request)
        return response

removing the middleware (which I'd like to keep though) solved the issue

@alramalho
Copy link

alramalho commented Mar 25, 2023

Same problem here, and I have a similar middleware as to @ClementWalte. (also with fastapi>=0.87.0)

alramalho added a commit to alramalho/ask-paper-ai that referenced this issue Mar 25, 2023
@RVERJUSAdeo
Copy link

Hi I have a similar problem which appears when fastapi > 0.90.0 and starlette > 0.22. I am using python 3.10 and uvicorn: 0.21.1.

@Kludex Kludex reopened this Mar 30, 2023
@parthvnp
Copy link
Contributor

parthvnp commented May 7, 2023

@Kludex Happy to investigate this issue if it still needs attention!

@bianucci
Copy link

bianucci commented May 8, 2023

Hi, same here.

Seeing this occasionally when applying a load test on localhost.

Using 5 Postman Runners, executed in parallel on same host.

uvicorn==0.22.0
fastapi==0.95.1

@mighty-phoenix
Copy link

Hey! @bianucci
I'm also facing the same issue in the same scenario.
I'm using locust to perform a load test on localhost.

@parthvnp
Copy link
Contributor

@mighty-phoenix @bianucci Can you please provide the complete stack trace as well the config for any other tools you might be using to performing load testing.

@sboudouk
Copy link

Same issue here with:

fastapi 0.95.1 FastAPI framework, high performance, easy to learn, fast to code, ready for production
├── pydantic >=1.6.2,<1.7 || >1.7,<1.7.1 || >1.7.1,<1.7.2 || >1.7.2,<1.7.3 || >1.7.3,<1.8 || >1.8,<1.8.1 || >1.8.1,<2.0.0
│   └── typing-extensions >=4.2.0 
└── starlette >=0.26.1,<0.27.0
    ├── anyio >=3.4.0,<5 
    │   ├── idna >=2.8 
    │   └── sniffio >=1.1 
    └── typing-extensions >=3.10.0 
    

@Kludex
Copy link
Sponsor Member

Kludex commented Jun 10, 2023

I'm going to lock this issue, since it should be a different cause from the initial issue.

Please open a new issue with an MRE (and reference this issue there).

I'll lock this discussion since people keep posting "same issue" messages.

@Kludex Kludex closed this as completed Jun 10, 2023
@encode encode locked as spam and limited conversation to collaborators Jun 10, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests