Cannot write to closing transport

[ajay1mg]

🐞 Describe the bug
Every few minutes we get Cannot write to closing transport exception on our gateway service which is a part of our microservices infrastructure.

Gateway service calls all other services with a timeout of 10 seconds. The other service that gateway calls are also based on aiohttp framework. One such service which is giving most of the exceptions doesn't raise any on its own server.

We are on ubuntu 16.04 LTS

💡 Expected behavior
Our gateway service should be able to handle the timeouts.

We are getting the following exception on our server every few minutes:
📋 Logs/tracebacks

Traceback (most recent call last):
  File "/home/ubuntu/.pyenv/versions/gateway/lib/python3.7/site-packages/aiohttp/client.py", line 502, in _request
    resp = await req.send(conn)
  File "/home/ubuntu/.pyenv/versions/gateway/lib/python3.7/site-packages/aiohttp/client_reqrep.py", line 629, in send
    await writer.write_headers(status_line, self.headers)
  File "/home/ubuntu/.pyenv/versions/gateway/lib/python3.7/site-packages/aiohttp/http_writer.py", line 112, in write_headers
    self._write(buf)
  File "/home/ubuntu/.pyenv/versions/gateway/lib/python3.7/site-packages/aiohttp/http_writer.py", line 67, in _write
    raise ConnectionResetError('Cannot write to closing transport')
ConnectionResetError: Cannot write to closing transport

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/ubuntu/.pyenv/versions/gateway/lib/python3.7/site-packages/hydra/base_api_request.py", line 75, in request
    timeout=cls.request_timeout(timeout))
  File "<string>", line 5, in wrapper
  File "/home/ubuntu/.pyenv/versions/3.7.2/envs/gateway/lib/python3.7/site-packages/newrelic/hooks/framework_aiohttp.py", line 260, in _coro
    response = yield from wrapped(*args, **kwargs)
  File "/home/ubuntu/.pyenv/versions/gateway/lib/python3.7/site-packages/aiohttp/client.py", line 514, in _request
    raise ClientOSError(*exc.args) from exc
aiohttp.client_exceptions.ClientOSError: Cannot write to closing transport

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ubuntu/.pyenv/versions/gateway/lib/python3.7/site-packages/hydra/decorators/http.py", line 41, in f
    result = await wait_for(shield(wrapped_func(self, *args, **kwargs)), api_timeout)
  File "/home/ubuntu/.pyenv/versions/3.7.2/lib/python3.7/asyncio/tasks.py", line 416, in wait_for
    return fut.result()
  File "/home/ubuntu/1mg/API-Gateway/gateway/utils.py", line 189, in wrapper
    result = await func(*args, **kwargs)
  File "/home/ubuntu/1mg/API-Gateway/gateway/managers/app_context_manager.py", line 98, in _build_shared_context
    result = await func(*args, **kwargs)
  File "/home/ubuntu/1mg/API-Gateway/gateway/handlers/mingler/service_handler.py", line 14, in services
    response = await Services.get_services(params)
  File "/home/ubuntu/1mg/API-Gateway/gateway/service_clients/mingler/services.py", line 13, in get_services
    result = await cls.post(path, data=params, headers=headers)
  File "/home/ubuntu/.pyenv/versions/gateway/lib/python3.7/site-packages/hydra/base_api_request.py", line 144, in post
    response_headers_list=response_headers_list)
  File "/home/ubuntu/.pyenv/versions/gateway/lib/python3.7/site-packages/hydra/base_api_request.py", line 93, in request
    raise HTTPRequestException(error={'message': exception_message})
hydra.exceptions.HTTPRequestException

📋 Your version of the Python

3.7.2

📋 Your version of the ubuntu/aiohttp/yarl/multidict distributions

aiohttp==3.6.2

multidict==4.5.2

yarl==1.3.0

📋 Additional context

This issue is already tracked in a number of threads on issue tracker on GitHub like:
ITISFoundation/osparc-simcore#1100
#3648

[arnoldfranz]

Thank you Dahuage for your fix, I will be happy to see it applied on a next release.

[Arnaud-Francois-Fausse]

Hi, Is it possible to by-pass this issue by a slight change in my application code ? Is it linked to the way I manage the session ?
Can anybody explain the root cause ?
Thanks
AF

[WoodyFleurant]

@Arnaud-Francois-Fausse From my understanding some HTTP requests that are closing (because eventually request failed ?) are reused for other requests, but as the Transport is 'Closing' it raise that Exception. @arnoldfranz made a fix to "reuse" only requests that are not using 'Closing' transport. On my end I tested using HTTP/1.0 with a global session and I did not faced that exception, but HTTP Frame were not reused.

[Dahuage]

Hi, Is it possible to by-pass this issue by a slight change in my application code ? Is it linked to the way I manage the session ?
Can anybody explain the root cause ?
Thanks
AF

Hope the explaination helps!

[asvetlov]

Fixed by mentioned PRs

[danielfaust]

I just wanted to leave a note saying that when I have a WebSocket connection open from a browser to an aiohttp server, and then kill that browser via the Task Manager (Windows), then the same error (ConnectionResetError: Cannot write to closing transport) will get raised.

Every following attempt to write to that WebSocketResponse object will raise the same error. When will it effectively close?

[asvetlov]

If a peer (a browser in your case) closes WebSocket -- the server cannot send data through it.
That's why any writing to the closed socket raises an exception.
The exception is caught by aiohttp server internals, a message about the unhandled exception is logged.
You can just ignore such records in logs but if you want tidy-up the code you can catch the exception explicitly and immediately exit from a web-handler.