openai.InternalServerError: upstream connect error or disconnect/reset before headers. reset reason: connection failure, transport failure reason: delayed connect error: 111

[JustAnotherVeryNormalDeveloper]

Confirm this is an issue with the Python library and not an underlying OpenAI API

• This is an issue with the Python library

Describe the bug

I am doing a hundreds of requests by hour, 95% works perfectly.
Sometimes it crashes with this error:
openai.InternalServerError: upstream connect error or disconnect/reset before headers. reset reason: connection failure, transport failure reason: delayed connect error: 111

The thing is that the same code works most of the time and fails sometime, making the debugging really hard, I failed to find the pattern that make theses requests fail, it can arrive anytime.

It can pass with big requests, with a lot of requests at the same time. It can crash with tiny request and with few requests in the same time.
So the intuition that they are too much request or too big request seems wrong.

I have this Traceback:

return func(*args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^\n  File "/usr/local/lib/python3.11/site-packages/openai/resources/chat/completions.py", line 650, in create\n    return self._post(\n           ^^^^^^^^^^^\n  File "/usr/local/lib/python3.11/site-packages/openai/_base_client.py", line 1259, in post\n    return cast(ResponseT, self.request(cast_to, opts, stream=stream, stream_cls=stream_cls))\n                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/usr/local/lib/python3.11/site-packages/openai/_base_client.py", line 936, in request\n    return self._request(\n           ^^^^^^^^^^^^^^\n  File "/usr/local/lib/python3.11/site-packages/openai/_base_client.py", line 1025, in _request\n    return self._retry_request(\n           ^^^^^^^^^^^^^^^^^^^^\n  File "/usr/local/lib/python3.11/site-packages/openai/_base_client.py", line 1074, in _retry_request\n    return self._request(\n           ^^^^^^^^^^^^^^\n  File "/usr/local/lib/python3.11/site-packages/openai/_base_client.py", line 1025, in _request\n    return self._retry_request(\n           ^^^^^^^^^^^^^^^^^^^^\n  File "/usr/local/lib/python3.11/site-packages/openai/_base_client.py", line 1074, in _retry_request\n    return self._request(\n           ^^^^^^^^^^^^^^\n  File "/usr/local/lib/python3.11/site-packages/openai/_base_client.py", line 1040, in _request\n    raise self._make_status_error_from_response(err.response) from None\nopenai.InternalServerError: upstream connect error or disconnect/reset before headers. reset reason: connection failure, transport failure reason: delayed connect error: 111\n

To Reproduce

from langchain_openai import ChatOpenAI

llm = ChatOpenAI(
model="gpt-4o",
temperature=0,
max_tokens=None,
timeout=None,
max_retries=2,
)

Code snippets

No response

OS

Linux

Python version

python v3.11

Library version

openai v1.35.10

[RobertCraigie]

what versions of httpx and httpcore do you have installed?

[JustAnotherVeryNormalDeveloper]

what versions of httpx and httpcore do you have installed?

httpx==0.27.0
httpcore==1.0.5

[RobertCraigie]

thanks, can you enable debug logging and share the response headers from a failed request?

you could also do something like this if it's easier

from openai import InternalServerError

try:
  # ... make request ...
except InternalServerError as exc:
  print(exc.response.headers)
  raise exc

[JustAnotherVeryNormalDeveloper]

So i added this:
except InternalServerError as exc:
print(f'InternalServerError error: {repr(exc)}')
print(f'InternalServerError error response: {repr(exc.response)}')
print(f'{format_log_type(self.name)} InternalServerError error headers: {repr(exc.response.headers)}')
raise exc

I receive this sometime:
InternalServerError error: InternalServerError('upstream connect error or disconnect/reset before headers. reset reason: connection failure, transport failure reason: delayed connect error: 111')
InternalServerError error response: <Response [503 Service Unavailable]>
InternalServerError error headers: Headers({'content-length': '145', 'content-type': 'text/plain', 'date': 'Fri, 09 Aug 2024 12:20:02 GMT', 'server': 'istio-envoy', 'connection': 'close', 'x-envoy-decorator-operation': 'tgi-cohere.xxxxxxx.svc.cluster.local:80/*'})

[RobertCraigie]

Thanks for the extra details. It looks like this is an intermittent network issue so it's very unlikely to be related to the SDK so I'm going to close this.

Are you using a proxy? Could there be an issue with that server?

If you need more assistance please use the community forum.

[JustAnotherVeryNormalDeveloper]

yes, I think there is not much more we can do from here, thanks for the help, I'm going to look from server side.