HTTP 502 errors on meta/musicgen long requests

[tigger0jk]

Some predictions succeed for me, i.e. https://replicate.com/black-forest-labs/flux-1.1-pro is fine.
But this one fails for me when I give it a longer duration to generate, i.e. 90 seconds of music, 502s after ~100-130 seconds or so maybe 90%+ of the time. https://replicate.com/meta/musicgen

This started failing for me around 2-3 hours ago.

After it fails, the request is still running in replicate backend, and will go on to ultimately succeed.

This happens to me on replicate 0.34.1 and 1.0.4. I had a very similar issue before, my use case and test case is basically the same as it was then: #372
The only real difference from my perspective is:

1. This happens to me regardless of version (last time the problem was introduced in 1.0.1), so I could workaround by downgrading
2. The problem was introduced through no changes on my end (the existing live version of my code simply started failing)

import os, replicate, time

# set your replicate API key somehow
os.environ["REPLICATE_API_TOKEN"] = REPLICATE_API_KEY

prompt = "Upbeat indie pop track with cheerful melodies and light percussion."
# note these do _cost_ more the longer you pass, and the request does execute
# usually succeeds in 12-13s, sometimes still 502s
# duration = 1
# 502s sometimes
# duration = 8
# 502s 90%+ of the time
duration = 90  # ~$0.25
# duration = 300  # ~$0.80
input = {
    "prompt": prompt,
    "model_version": "stereo-large",
    "duration": duration,
    "output_format": "mp3",
}
replicate_model = (
    "meta/musicgen:671ac645ce5e552cc63a54a2bbff63fcf798043055d2dac5fc9e36a837eedcfb"
)

start_time = time.time()
try:
    output = replicate.run(replicate_model, input=input)
    print(output)
except Exception as e:
    print(e)
    print("error generating music: " + str(e))
    print(type(e))
    print(e.__dict__)
end_time = time.time()
print(f"Request took {end_time - start_time:.2f} seconds")

And the output

python3 replicate_error.py
ReplicateError Details:
status: 502
error generating music: ReplicateError Details:
status: 502
<class 'replicate.exceptions.ReplicateError'>
{'type': None, 'title': None, 'status': 502, 'detail': None, 'instance': None}
Request took 135.18 seconds

[nickstenning]

This is most likely related to the outage we experienced yesterday. Are you still having problems?

[tigger0jk]

Thanks for the update, this is working fine for me now.