Hang in OpenAI package with many threads -- not thread safe?

[pseudotensor]

I'm using gradio package -> OpenAI API package -> network to vLLM. However, httpx hangs frequently, so why I'm here.

The only possible explanation to me is that httpx or httpcore are not thread safe.

The setup is that gradio uses asyncio and I launch threads for each chatbot like at https://gpt.h2o.ai . Typically the problem is with the most-used chatbot. There can be many users, each user's chatbot would have a thread. However, while things work for a week or so without any issue with heavy use, for some reason at some point connection timeouts.

Key is that after this starts, not a single user or connection can connect to a particular vLLM IP. It just keeps on timing out forever from then on.

WARNING:  Invalid HTTP request received.
Traceback (most recent call last):
  File "/home/ubuntu/miniconda3/envs/h2ogpt/lib/python3.10/site-packages/httpcore/_exceptions.py", line 10, in map_exceptions
    yield
  File "/home/ubuntu/miniconda3/envs/h2ogpt/lib/python3.10/site-packages/httpcore/_backends/sync.py", line 126, in read
    return self._sock.recv(max_bytes)
TimeoutError: timed out

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/ubuntu/miniconda3/envs/h2ogpt/lib/python3.10/site-packages/httpx/_transports/default.py", line 66, in map_httpcore_exceptions
    yield
  File "/home/ubuntu/miniconda3/envs/h2ogpt/lib/python3.10/site-packages/httpx/_transports/default.py", line 110, in __iter__
    for part in self._httpcore_stream:
  File "/home/ubuntu/miniconda3/envs/h2ogpt/lib/python3.10/site-packages/httpcore/_sync/connection_pool.py", line 361, in __iter__
    for part in self._stream:
  File "/home/ubuntu/miniconda3/envs/h2ogpt/lib/python3.10/site-packages/httpcore/_sync/http11.py", line 337, in __iter__
    raise exc
  File "/home/ubuntu/miniconda3/envs/h2ogpt/lib/python3.10/site-packages/httpcore/_sync/http11.py", line 329, in __iter__
    for chunk in self._connection._receive_response_body(**kwargs):
  File "/home/ubuntu/miniconda3/envs/h2ogpt/lib/python3.10/site-packages/httpcore/_sync/http11.py", line 198, in _receive_response_body
    event = self._receive_event(timeout=timeout)
  File "/home/ubuntu/miniconda3/envs/h2ogpt/lib/python3.10/site-packages/httpcore/_sync/http11.py", line 212, in _receive_event
    data = self._network_stream.read(
  File "/home/ubuntu/miniconda3/envs/h2ogpt/lib/python3.10/site-packages/httpcore/_backends/sync.py", line 124, in read
    with map_exceptions(exc_map):
  File "/home/ubuntu/miniconda3/envs/h2ogpt/lib/python3.10/contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/home/ubuntu/miniconda3/envs/h2ogpt/lib/python3.10/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions
    raise to_exc(exc) from exc
httpcore.ReadTimeout: timed out

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/ubuntu/miniconda3/envs/h2ogpt/lib/python3.10/site-packages/gradio/queueing.py", line 407, in call_prediction
    output = await route_utils.call_process_api(
  File "/home/ubuntu/miniconda3/envs/h2ogpt/lib/python3.10/site-packages/gradio/route_utils.py", line 226, in call_process_api
    output = await app.get_blocks().process_api(
  File "/home/ubuntu/miniconda3/envs/h2ogpt/lib/python3.10/site-packages/gradio/blocks.py", line 1550, in process_api
    result = await self.call_function(
  File "/home/ubuntu/miniconda3/envs/h2ogpt/lib/python3.10/site-packages/gradio/blocks.py", line 1199, in call_function
    prediction = await utils.async_iteration(iterator)
  File "/home/ubuntu/miniconda3/envs/h2ogpt/lib/python3.10/site-packages/gradio/utils.py", line 519, in async_iteration
    return await iterator.__anext__()
  File "/home/ubuntu/miniconda3/envs/h2ogpt/lib/python3.10/site-packages/gradio/utils.py", line 512, in __anext__
    return await anyio.to_thread.run_sync(
  File "/home/ubuntu/miniconda3/envs/h2ogpt/lib/python3.10/site-packages/anyio/to_thread.py", line 33, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "/home/ubuntu/miniconda3/envs/h2ogpt/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 877, in run_sync_in_worker_thread
    return await future
  File "/home/ubuntu/miniconda3/envs/h2ogpt/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 807, in run
    result = context.run(func, *args)
  File "/home/ubuntu/miniconda3/envs/h2ogpt/lib/python3.10/site-packages/gradio/utils.py", line 495, in run_sync_iterator_async
    return next(iterator)
  File "/home/ubuntu/miniconda3/envs/h2ogpt/lib/python3.10/site-packages/gradio/utils.py", line 649, in gen_wrapper
    yield from f(*args, **kwargs)
  File "/home/ubuntu/fluidstack/h2ogpt.2/h2o-llm/src/gradio_runner.py", line 3526, in evaluate_nochat
    for res in get_response(fun1, history, chatbot_role1, speaker1, tts_language1, roles_state1,
  File "/home/ubuntu/fluidstack/h2ogpt.2/h2o-llm/src/gradio_runner.py", line 4116, in get_response
    for output_fun in fun1():
  File "/home/ubuntu/fluidstack/h2ogpt.2/h2o-llm/src/gen.py", line 3816, in evaluate
    for event in responses:
  File "/home/ubuntu/miniconda3/envs/h2ogpt/lib/python3.10/site-packages/openvllm/_streaming.py", line 40, in __iter__
    for item in self._iterator:
  File "/home/ubuntu/miniconda3/envs/h2ogpt/lib/python3.10/site-packages/openvllm/_streaming.py", line 52, in __stream__
    for sse in iterator:
  File "/home/ubuntu/miniconda3/envs/h2ogpt/lib/python3.10/site-packages/openvllm/_streaming.py", line 44, in _iter_events
    yield from self._decoder.iter(self.response.iter_lines())
  File "/home/ubuntu/miniconda3/envs/h2ogpt/lib/python3.10/site-packages/openvllm/_streaming.py", line 182, in iter
    for line in iterator:
  File "/home/ubuntu/miniconda3/envs/h2ogpt/lib/python3.10/site-packages/httpx/_models.py", line 862, in iter_lines
    for text in self.iter_text():
  File "/home/ubuntu/miniconda3/envs/h2ogpt/lib/python3.10/site-packages/httpx/_models.py", line 849, in iter_text
    for byte_content in self.iter_bytes():
  File "/home/ubuntu/miniconda3/envs/h2ogpt/lib/python3.10/site-packages/httpx/_models.py", line 828, in iter_bytes
    for raw_bytes in self.iter_raw():
  File "/home/ubuntu/miniconda3/envs/h2ogpt/lib/python3.10/site-packages/httpx/_models.py", line 886, in iter_raw
    for raw_stream_bytes in self.stream:
  File "/home/ubuntu/miniconda3/envs/h2ogpt/lib/python3.10/site-packages/httpx/_client.py", line 123, in __iter__
    for chunk in self._stream:
  File "/home/ubuntu/miniconda3/envs/h2ogpt/lib/python3.10/site-packages/httpx/_transports/default.py", line 109, in __iter__
    with map_httpcore_exceptions():
  File "/home/ubuntu/miniconda3/envs/h2ogpt/lib/python3.10/contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/home/ubuntu/miniconda3/envs/h2ogpt/lib/python3.10/site-packages/httpx/_transports/default.py", line 83, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.ReadTimeout: timed out
Traceback (most recent call last):
  File "/home/ubuntu/miniconda3/envs/h2ogpt/lib/python3.10/site-packages/httpcore/_exceptions.py", line 10, in map_exceptions
    yield
  File "/home/ubuntu/miniconda3/envs/h2ogpt/lib/python3.10/site-packages/httpcore/_backends/sync.py", line 126, in read
    return self._sock.recv(max_bytes)
TimeoutError: timed out

So there are many many of these kinds of blocks like above.

Once this is hit, I go back and then wonder why it's stuck forever, since every use of OpenAI is fresh. That is, I create a fresh client every time, so I don't understand why things are stuck since nothing is re-used on my end.

Once this is happening, I go back and check the threads in the process (I have a faulthandler attached), then I see numerous threads like this the below. This is with 0 use of the the process by any user at this point (it's been isolated).

Thread 0x00007f8930cfb640 (most recent call first):
  File "/home/ubuntu/miniconda3/envs/h2ogpt/lib/python3.10/threading.py", line 324 in wait
  File "/home/ubuntu/miniconda3/envs/h2ogpt/lib/python3.10/threading.py", line 607 in wait
  File "/home/ubuntu/miniconda3/envs/h2ogpt/lib/python3.10/site-packages/httpcore/_synchronization.py", line 231 in wait
  File "/home/ubuntu/miniconda3/envs/h2ogpt/lib/python3.10/site-packages/httpcore/_sync/connection_pool.py", line 36 in wait_for_connection
  File "/home/ubuntu/miniconda3/envs/h2ogpt/lib/python3.10/site-packages/httpcore/_sync/connection_pool.py", line 239 in handle_request
  File "/home/ubuntu/miniconda3/envs/h2ogpt/lib/python3.10/site-packages/httpx/_transports/default.py", line 228 in handle_request
  File "/home/ubuntu/miniconda3/envs/h2ogpt/lib/python3.10/site-packages/httpx/_client.py", line 1002 in _send_single_request
  File "/home/ubuntu/miniconda3/envs/h2ogpt/lib/python3.10/site-packages/httpx/_client.py", line 966 in _send_handling_redirects
  File "/home/ubuntu/miniconda3/envs/h2ogpt/lib/python3.10/site-packages/httpx/_client.py", line 929 in _send_handling_auth
  File "/home/ubuntu/miniconda3/envs/h2ogpt/lib/python3.10/site-packages/httpx/_client.py", line 901 in send
  File "/home/ubuntu/miniconda3/envs/h2ogpt/lib/python3.10/site-packages/openai/_base_client.py", line 882 in _request
  File "/home/ubuntu/miniconda3/envs/h2ogpt/lib/python3.10/site-packages/openai/_base_client.py", line 966 in _retry_request
  File "/home/ubuntu/miniconda3/envs/h2ogpt/lib/python3.10/site-packages/openai/_base_client.py", line 914 in _request
  File "/home/ubuntu/miniconda3/envs/h2ogpt/lib/python3.10/site-packages/openai/_base_client.py", line 966 in _retry_request
  File "/home/ubuntu/miniconda3/envs/h2ogpt/lib/python3.10/site-packages/openai/_base_client.py", line 914 in _request
  File "/home/ubuntu/miniconda3/envs/h2ogpt/lib/python3.10/site-packages/openai/_base_client.py", line 856 in request
  File "/home/ubuntu/miniconda3/envs/h2ogpt/lib/python3.10/site-packages/openai/_base_client.py", line 1096 in post
  File "/home/ubuntu/miniconda3/envs/h2ogpt/lib/python3.10/site-packages/openai/resources/completions.py", line 563 in create
  File "/home/ubuntu/miniconda3/envs/h2ogpt/lib/python3.10/site-packages/openai/_utils/_utils.py", line 301 in wrapper
  File "/home/ubuntu/fluidstack/h2ogpt.2/h2o-llm/src/gen.py", line 3801 in evaluate
  File "/home/ubuntu/fluidstack/h2ogpt.2/h2o-llm/src/gradio_runner.py", line 4116 in get_response
  File "/home/ubuntu/fluidstack/h2ogpt.2/h2o-llm/src/gradio_runner.py", line 3526 in evaluate_nochat
  File "/home/ubuntu/miniconda3/envs/h2ogpt/lib/python3.10/site-packages/gradio/utils.py", line 649 in gen_wrapper
  File "/home/ubuntu/miniconda3/envs/h2ogpt/lib/python3.10/site-packages/gradio/utils.py", line 495 in run_sync_iterator_async
  File "/home/ubuntu/miniconda3/envs/h2ogpt/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 807 in run
  File "/home/ubuntu/miniconda3/envs/h2ogpt/lib/python3.10/threading.py", line 1016 in _bootstrap_inner
  File "/home/ubuntu/miniconda3/envs/h2ogpt/lib/python3.10/threading.py", line 973 in _bootstrap

It's as if prior timeouts are making everything stuck in the same synchronization during handle_request in httpx.

I don't understand what could cause a single connection to a single IP out of many IPs (other chatbots are totally fine) to be blocked like this.

The only possible explanation to me is that httpx or httpcore are not thread safe.

[pseudotensor]

If there are any debug suggestions, the program is still up at moment in this bad state.

[zanieb]

Generally I'd recommend using py-spy to attach to the process and see what it's up to

https://github.com/benfred/py-spy

[zanieb]

I'm not sure I'm entirely clear on your setup, but asynchronous Python is not intended to be used with threads. If you mix threads and asynchronous code, you're very likely to encounter deadlocks e.g. a few scenarios

• If you're using one event loop across multiple threads, things will break quite quickly as the standard library async synchronization primitives are not threadsafe.
• If you're using synchronous code e.g. a thread from within an asynchronous context without properly unblocking the event loop, you'll see hangs in all other asynchronous tasks.
• If you're using multiple event loops, each isolated to a single thread, you'll be okay but the asynchronous performance will be bad as the event loop does not exist to be interrupted by synchronous context switches.