http read operation times out waiting for long generation on 1.0.1

[tigger0jk]

I just upgraded to replicate python 1.0.1 (live on PyPI but not github?) and started getting timeout errors, downgrading to 0.34.1 fixes it for me.

After upgrading, some predictions succeed for me, i.e. https://replicate.com/black-forest-labs/flux-1.1-pro is fine.
But one long one times out for me, https://replicate.com/meta/musicgen with a longer duration, i.e. 90+ seconds almost always fails.

Here is some example code:

import os, replicate

# set your replicate API key somehow
os.environ["REPLICATE_API_TOKEN"] = REPLICATE_API_KEY

prompt = "Upbeat indie pop track with cheerful melodies and light percussion."
# note these do _cost_ more the longer you pass, and the request does execute
#  duration = 8   # succeeds
#  duration = 90  # ~$0.25 times out sometimes?
duration = 300 # ~$0.80 times out consistently?
input = {
    "prompt": prompt,
    "model_version": "stereo-large",
    "duration": duration,
    "output_format": "mp3",
}
replicate_model = "meta/musicgen:671ac645ce5e552cc63a54a2bbff63fcf798043055d2dac5fc9e36a837eedcfb"

output = replicate.run(replicate_model, input=input)
print(output)

And the output:

Traceback (most recent call last):
  File "/Users/pfay/code/plusone/video-gen/venv/lib/python3.11/site-packages/httpx/_transports/default.py", line 72, in map_httpcore_exceptions
    yield
  File "/Users/pfay/code/plusone/video-gen/venv/lib/python3.11/site-packages/httpx/_transports/default.py", line 236, in handle_request
    resp = self._pool.handle_request(req)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/pfay/code/plusone/video-gen/venv/lib/python3.11/site-packages/httpcore/_sync/connection_pool.py", line 216, in handle_request
    raise exc from None
  File "/Users/pfay/code/plusone/video-gen/venv/lib/python3.11/site-packages/httpcore/_sync/connection_pool.py", line 196, in handle_request
    response = connection.handle_request(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/pfay/code/plusone/video-gen/venv/lib/python3.11/site-packages/httpcore/_sync/connection.py", line 101, in handle_request
    return self._connection.handle_request(request)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/pfay/code/plusone/video-gen/venv/lib/python3.11/site-packages/httpcore/_sync/http11.py", line 143, in handle_request
    raise exc
  File "/Users/pfay/code/plusone/video-gen/venv/lib/python3.11/site-packages/httpcore/_sync/http11.py", line 113, in handle_request
    ) = self._receive_response_headers(**kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/pfay/code/plusone/video-gen/venv/lib/python3.11/site-packages/httpcore/_sync/http11.py", line 186, in _receive_response_headers
    event = self._receive_event(timeout=timeout)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/pfay/code/plusone/video-gen/venv/lib/python3.11/site-packages/httpcore/_sync/http11.py", line 224, in _receive_event
    data = self._network_stream.read(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/pfay/code/plusone/video-gen/venv/lib/python3.11/site-packages/httpcore/_backends/sync.py", line 124, in read
    with map_exceptions(exc_map):
  File "/Users/pfay/.pyenv/versions/3.11.9/lib/python3.11/contextlib.py", line 158, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/Users/pfay/code/plusone/video-gen/venv/lib/python3.11/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions
    raise to_exc(exc) from exc
httpcore.ReadTimeout: The read operation timed out

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/pfay/code/plusone/video-gen/replicate_error.py", line 22, in <module>
    output = replicate.run(replicate_model, input=input)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/pfay/code/plusone/video-gen/venv/lib/python3.11/site-packages/replicate/client.py", line 175, in run
    return run(self, ref, input, use_file_output=use_file_output, **params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/pfay/code/plusone/video-gen/venv/lib/python3.11/site-packages/replicate/run.py", line 47, in run
    prediction = client.predictions.create(
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/pfay/code/plusone/video-gen/venv/lib/python3.11/site-packages/replicate/prediction.py", line 494, in create
    resp = self._client._request(
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/pfay/code/plusone/video-gen/venv/lib/python3.11/site-packages/replicate/client.py", line 88, in _request
    resp = self._client.request(method, path, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/pfay/code/plusone/video-gen/venv/lib/python3.11/site-packages/httpx/_client.py", line 837, in request
    return self.send(request, auth=auth, follow_redirects=follow_redirects)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/pfay/code/plusone/video-gen/venv/lib/python3.11/site-packages/httpx/_client.py", line 926, in send
    response = self._send_handling_auth(
               ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/pfay/code/plusone/video-gen/venv/lib/python3.11/site-packages/httpx/_client.py", line 954, in _send_handling_auth
    response = self._send_handling_redirects(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/pfay/code/plusone/video-gen/venv/lib/python3.11/site-packages/httpx/_client.py", line 991, in _send_handling_redirects
    response = self._send_single_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/pfay/code/plusone/video-gen/venv/lib/python3.11/site-packages/httpx/_client.py", line 1027, in _send_single_request
    response = transport.handle_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/pfay/code/plusone/video-gen/venv/lib/python3.11/site-packages/replicate/client.py", line 293, in handle_request
    response = self._wrapped_transport.handle_request(request)  # type: ignore
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/pfay/code/plusone/video-gen/venv/lib/python3.11/site-packages/httpx/_transports/default.py", line 235, in handle_request
    with map_httpcore_exceptions():
  File "/Users/pfay/.pyenv/versions/3.11.9/lib/python3.11/contextlib.py", line 158, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/Users/pfay/code/plusone/video-gen/venv/lib/python3.11/site-packages/httpx/_transports/default.py", line 89, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.ReadTimeout: The read operation timed out

[asppj]

+1 httpcore.ReadTimeout: The read operation timed out

replicate                1.0.1

[asppj]

Return to 1.0.0 and it can be used normally

pip install replicate==1.0.0

[paulocoutinhox]

I have the same problem. Can someone fix this asap? Thanks guys.

[aron]

This should now be fixed in 1.0.2. Please let us know if you run into further issues.

[tigger0jk]

I had been pinned to replicate 0.34.1 since this issue, and today without any changes on my end started getting pretty consistent 502 errors (90%+) for the same case as this test script, starting around 2-3 hours ago. Upgrading to replicate 1.0.4 does not fix this.

Other issue: #402