Event loop is closed error when running with global client

[ryancausey]

Hello,

I am developing a function for AWS lambda, and I'm using httpx with asyncio. I'm running into an issue when using the code from the python REPL whereby re-running the lambda_handler function after successfully running it leads to a "RuntimeError: Event loop is closed" exception. I believe this is due to following the "single global client instance" from the note in the docs while calling asyncio.run(main()) from the handler.

Is this expected behavior? I was hoping to follow the pattern of having a global client instance so that connection pools would be re-used across multiple requests while the lambda execution environment was warm. I've used this same pattern successfully in sync code without issue.

If I instead create the client inside async def main() each time, the error no longer occurs within the REPL.

Minimal reproduction:

# pdf_processing/test_httpx_aio.py
import asyncio

import httpx

http_client = httpx.AsyncClient()


async def main():
    async with asyncio.TaskGroup() as task_group:
        for _ in range(10):
            task_group.create_task(http_client.get(url="https://example.com"))


def lambda_handler(event, context):
    asyncio.run(main())

HTTPX version: httpx = "0.27.0"

Results running in the REPL:

expand for shell output

$ poetry run python
The currently activated Python version 3.9.6 is not supported by the project (^3.11).
Trying to find and use a compatible version.
Using python3 (3.11.4)
Python 3.11.4 (main, Jul 20 2023, 18:33:17) [GCC 11.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from pdf_processing.test_httpx_aio import lambda_handler
>>> lambda_handler(None, None)
>>> lambda_handler(None, None)
  + Exception Group Traceback (most recent call last):
  |   File "<stdin>", line 1, in <module>
  |   File "/home/rccausey/workspace/munipal/notification-services/notifications-api/pdf_processing/test_httpx_aio.py", line 15, in lambda_handler
  |     asyncio.run(main())
  |   File "/home/rccausey/.pyenv/versions/3.11.4/lib/python3.11/asyncio/runners.py", line 190, in run
  |     return runner.run(main)
  |            ^^^^^^^^^^^^^^^^
  |   File "/home/rccausey/.pyenv/versions/3.11.4/lib/python3.11/asyncio/runners.py", line 118, in run
  |     return self._loop.run_until_complete(task)
  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/home/rccausey/.pyenv/versions/3.11.4/lib/python3.11/asyncio/base_events.py", line 653, in run_until_complete
  |     return future.result()
  |            ^^^^^^^^^^^^^^^
  |   File "/home/rccausey/workspace/munipal/notification-services/notifications-api/pdf_processing/test_httpx_aio.py", line 9, in main
  |     async with asyncio.TaskGroup() as task_group:
  |   File "/home/rccausey/.pyenv/versions/3.11.4/lib/python3.11/asyncio/taskgroups.py", line 147, in __aexit__
  |     raise me from None
  | ExceptionGroup: unhandled errors in a TaskGroup (2 sub-exceptions)
  +-+---------------- 1 ----------------
    | Traceback (most recent call last):
    |   File "/home/rccausey/.cache/pypoetry/virtualenvs/notifications-api-6mUc-8uv-py3.11/lib/python3.11/site-packages/httpx/_client.py", line 1801, in get
    |     return await self.request(
    |            ^^^^^^^^^^^^^^^^^^^
    |   File "/home/rccausey/.cache/pypoetry/virtualenvs/notifications-api-6mUc-8uv-py3.11/lib/python3.11/site-packages/httpx/_client.py", line 1574, in request
    |     return await self.send(request, auth=auth, follow_redirects=follow_redirects)
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/home/rccausey/.cache/pypoetry/virtualenvs/notifications-api-6mUc-8uv-py3.11/lib/python3.11/site-packages/httpx/_client.py", line 1661, in send
    |     response = await self._send_handling_auth(
    |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/home/rccausey/.cache/pypoetry/virtualenvs/notifications-api-6mUc-8uv-py3.11/lib/python3.11/site-packages/httpx/_client.py", line 1689, in _send_handling_auth
    |     response = await self._send_handling_redirects(
    |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/home/rccausey/.cache/pypoetry/virtualenvs/notifications-api-6mUc-8uv-py3.11/lib/python3.11/site-packages/httpx/_client.py", line 1726, in _send_handling_redirects
    |     response = await self._send_single_request(request)
    |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/home/rccausey/.cache/pypoetry/virtualenvs/notifications-api-6mUc-8uv-py3.11/lib/python3.11/site-packages/httpx/_client.py", line 1763, in _send_single_request
    |     response = await transport.handle_async_request(request)
    |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/home/rccausey/.cache/pypoetry/virtualenvs/notifications-api-6mUc-8uv-py3.11/lib/python3.11/site-packages/httpx/_transports/default.py", line 373, in handle_async_request
    |     resp = await self._pool.handle_async_request(req)
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/home/rccausey/.cache/pypoetry/virtualenvs/notifications-api-6mUc-8uv-py3.11/lib/python3.11/site-packages/httpcore/_async/connection_pool.py", line 216, in handle_async_request
    |     raise exc from None
    |   File "/home/rccausey/.cache/pypoetry/virtualenvs/notifications-api-6mUc-8uv-py3.11/lib/python3.11/site-packages/httpcore/_async/connection_pool.py", line 196, in handle_async_request
    |     response = await connection.handle_async_request(
    |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/home/rccausey/.cache/pypoetry/virtualenvs/notifications-api-6mUc-8uv-py3.11/lib/python3.11/site-packages/httpcore/_async/connection.py", line 101, in handle_async_request
    |     return await self._connection.handle_async_request(request)
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/home/rccausey/.cache/pypoetry/virtualenvs/notifications-api-6mUc-8uv-py3.11/lib/python3.11/site-packages/httpcore/_async/http11.py", line 142, in handle_async_request
    |     await self._response_closed()
    |   File "/home/rccausey/.cache/pypoetry/virtualenvs/notifications-api-6mUc-8uv-py3.11/lib/python3.11/site-packages/httpcore/_async/http11.py", line 257, in _response_closed
    |     await self.aclose()
    |   File "/home/rccausey/.cache/pypoetry/virtualenvs/notifications-api-6mUc-8uv-py3.11/lib/python3.11/site-packages/httpcore/_async/http11.py", line 265, in aclose
    |     await self._network_stream.aclose()
    |   File "/home/rccausey/.cache/pypoetry/virtualenvs/notifications-api-6mUc-8uv-py3.11/lib/python3.11/site-packages/httpcore/_backends/anyio.py", line 55, in aclose
    |     await self._stream.aclose()
    |   File "/home/rccausey/.cache/pypoetry/virtualenvs/notifications-api-6mUc-8uv-py3.11/lib/python3.11/site-packages/anyio/streams/tls.py", line 202, in aclose
    |     await self.transport_stream.aclose()
    |   File "/home/rccausey/.cache/pypoetry/virtualenvs/notifications-api-6mUc-8uv-py3.11/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 1191, in aclose
    |     self._transport.close()
    |   File "/home/rccausey/.pyenv/versions/3.11.4/lib/python3.11/asyncio/selector_events.py", line 860, in close
    |     self._loop.call_soon(self._call_connection_lost, None)
    |   File "/home/rccausey/.pyenv/versions/3.11.4/lib/python3.11/asyncio/base_events.py", line 761, in call_soon
    |     self._check_closed()
    |   File "/home/rccausey/.pyenv/versions/3.11.4/lib/python3.11/asyncio/base_events.py", line 519, in _check_closed
    |     raise RuntimeError('Event loop is closed')
    | RuntimeError: Event loop is closed
    +---------------- 2 ----------------
    | Traceback (most recent call last):
    |   File "/home/rccausey/.cache/pypoetry/virtualenvs/notifications-api-6mUc-8uv-py3.11/lib/python3.11/site-packages/httpx/_client.py", line 1801, in get
    |     return await self.request(
    |            ^^^^^^^^^^^^^^^^^^^
    |   File "/home/rccausey/.cache/pypoetry/virtualenvs/notifications-api-6mUc-8uv-py3.11/lib/python3.11/site-packages/httpx/_client.py", line 1574, in request
    |     return await self.send(request, auth=auth, follow_redirects=follow_redirects)
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/home/rccausey/.cache/pypoetry/virtualenvs/notifications-api-6mUc-8uv-py3.11/lib/python3.11/site-packages/httpx/_client.py", line 1661, in send
    |     response = await self._send_handling_auth(
    |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/home/rccausey/.cache/pypoetry/virtualenvs/notifications-api-6mUc-8uv-py3.11/lib/python3.11/site-packages/httpx/_client.py", line 1689, in _send_handling_auth
    |     response = await self._send_handling_redirects(
    |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/home/rccausey/.cache/pypoetry/virtualenvs/notifications-api-6mUc-8uv-py3.11/lib/python3.11/site-packages/httpx/_client.py", line 1726, in _send_handling_redirects
    |     response = await self._send_single_request(request)
    |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/home/rccausey/.cache/pypoetry/virtualenvs/notifications-api-6mUc-8uv-py3.11/lib/python3.11/site-packages/httpx/_client.py", line 1763, in _send_single_request
    |     response = await transport.handle_async_request(request)
    |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/home/rccausey/.cache/pypoetry/virtualenvs/notifications-api-6mUc-8uv-py3.11/lib/python3.11/site-packages/httpx/_transports/default.py", line 373, in handle_async_request
    |     resp = await self._pool.handle_async_request(req)
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/home/rccausey/.cache/pypoetry/virtualenvs/notifications-api-6mUc-8uv-py3.11/lib/python3.11/site-packages/httpcore/_async/connection_pool.py", line 216, in handle_async_request
    |     raise exc from None
    |   File "/home/rccausey/.cache/pypoetry/virtualenvs/notifications-api-6mUc-8uv-py3.11/lib/python3.11/site-packages/httpcore/_async/connection_pool.py", line 196, in handle_async_request
    |     response = await connection.handle_async_request(
    |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/home/rccausey/.cache/pypoetry/virtualenvs/notifications-api-6mUc-8uv-py3.11/lib/python3.11/site-packages/httpcore/_async/connection.py", line 101, in handle_async_request
    |     return await self._connection.handle_async_request(request)
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/home/rccausey/.cache/pypoetry/virtualenvs/notifications-api-6mUc-8uv-py3.11/lib/python3.11/site-packages/httpcore/_async/http11.py", line 142, in handle_async_request
    |     await self._response_closed()
    |   File "/home/rccausey/.cache/pypoetry/virtualenvs/notifications-api-6mUc-8uv-py3.11/lib/python3.11/site-packages/httpcore/_async/http11.py", line 257, in _response_closed
    |     await self.aclose()
    |   File "/home/rccausey/.cache/pypoetry/virtualenvs/notifications-api-6mUc-8uv-py3.11/lib/python3.11/site-packages/httpcore/_async/http11.py", line 265, in aclose
    |     await self._network_stream.aclose()
    |   File "/home/rccausey/.cache/pypoetry/virtualenvs/notifications-api-6mUc-8uv-py3.11/lib/python3.11/site-packages/httpcore/_backends/anyio.py", line 55, in aclose
    |     await self._stream.aclose()
    |   File "/home/rccausey/.cache/pypoetry/virtualenvs/notifications-api-6mUc-8uv-py3.11/lib/python3.11/site-packages/anyio/streams/tls.py", line 202, in aclose
    |     await self.transport_stream.aclose()
    |   File "/home/rccausey/.cache/pypoetry/virtualenvs/notifications-api-6mUc-8uv-py3.11/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 1191, in aclose
    |     self._transport.close()
    |   File "/home/rccausey/.pyenv/versions/3.11.4/lib/python3.11/asyncio/selector_events.py", line 860, in close
    |     self._loop.call_soon(self._call_connection_lost, None)
    |   File "/home/rccausey/.pyenv/versions/3.11.4/lib/python3.11/asyncio/base_events.py", line 761, in call_soon
    |     self._check_closed()
    |   File "/home/rccausey/.pyenv/versions/3.11.4/lib/python3.11/asyncio/base_events.py", line 519, in _check_closed
    |     raise RuntimeError('Event loop is closed')
    | RuntimeError: Event loop is closed
    +------------------------------------
>>>

[tomchristie]

so that connection pools would be re-used across multiple requests while the lambda execution environment was warm.

Hm... I'd be curious what's actually occurring here. I'd be surprised if outgoing HTTP connections are being maintained across lambda invocations in most deployment environments.

I think the root of what's actually being highlighted here is that for the async case (where we're forced to be more explicit about I/O usage and resource lifetimes) we should probably gracefully transition towards only allowing the context-managed style for the client. (Eg. see also aiohttp).

Forcing this usage...

async with httpx.AsyncClient() as client:
    ...

Would ensure that async clients have explicitly clear lifetimes, and aren't used across different invocations of an event loop.

[ryancausey]

I'm going to be testing this inside lambda code today, so I'll see if I can reproduce this behavior within the lambda service itself.

However, if the whole "global client for cross-invocation connection pooling" idea is complete nonsense for both the sync and async case, I will probably just switch to the async with syntax and pass around the client instance.

[ryancausey]

Yep, same behavior in the lambda environment. So I'll switch to using the async with syntax.

[ryancausey]

I think the root of what's actually being highlighted here is that for the async case (where we're forced to be more explicit about I/O usage and resource lifetimes) we should probably gracefully transition towards only allowing the context-managed style for the client. (Eg. see also aiohttp).

@tomchristie based on this, did you want to turn this into an issue, or should I instead resolve this discussion?