Replicate only give me error 500

[paulocoutinhox]

Hi,

Im trying make a simple call for "meta-llama-3.1-405b-instruct" but it always return erro 500:

DEBUG 2024-10-23 05:49:13,367 _trace receive_response_headers.complete return_value=(b'HTTP/1.1', 500, b'Internal Server Error', [(b'Date', b'Wed, 23 Oct 2024 05:49:13 GMT'), (b'Content-Type', b'application/json; charset=UTF-8'), (b'Content-Length', b'41'), (b'Connection', b'keep-alive'), (b'Preference-Applied', b'wait'), (b'Report-To', b'{"endpoints":[{"url":"https:\\/\\/a.nel.cloudflare.com\\/report\\/v4?s=26t4irv%2Bd915fWOFYROvw%2BhoJFgA5aEquIeLbjWo0ZetlQOZ5bwRGa8wxmM53behG7hVKjDiP6NTGAD38EgI%2B7jip3KD%2FWamYc0Hj9th%2BeBi%2BCxWZaoYwQJ2HkveKno9YF6W"}],"group":"cf-nel","max_age":604800}'), (b'NEL', b'{"success_fraction":0,"report_to":"cf-nel","max_age":604800}'), (b'Vary', b'Accept-Encoding'), (b'Strict-Transport-Security', b'max-age=15552000'), (b'Server', b'cloudflare'), (b'CF-RAY', b'8d6f71069b60e12d-GIG'), (b'alt-svc', b'h3=":443"; ma=86400')])
INFO 2024-10-23 05:49:13,372 _client HTTP Request: POST https://api.replicate.com/v1/predictions "HTTP/1.1 500 Internal Server Error"
DEBUG 2024-10-23 05:49:13,373 _trace receive_response_body.started request=<Request [b'POST']>
DEBUG 2024-10-23 05:49:13,374 _trace receive_response_body.complete
DEBUG 2024-10-23 05:49:13,374 _trace response_closed.started
DEBUG 2024-10-23 05:49:13,374 _trace response_closed.complete
ERROR 2024-10-23 05:49:13,374 flux Replicate error: ReplicateError Details:
status: 500
detail: An unexpected error occurred

My code:

model = replicate.models.get("meta", "meta-llama-3.1-405b-instruct")

prompt_input = "say hello in python"

result = replicate.run(
    model.latest_version,
    input={
        "top_k": 50,
        "top_p": 0.9,
        "prompt": prompt_input,
        "max_tokens": 1024,
        "min_tokens": 0,
        "temperature": 0.6,
        "presence_penalty": 0,
        "frequency_penalty": 0,
    },
)

When i see in Replicate dashboard the request/response is there and the data is generated. But it is not returned when call the API returning error 500.

Im using lib replicate==1.0.2.

What is wrong?

Thanks.

[nickstenning]

Thank you for the report! The 500 error is a problem on our side which we'll work on fixing separately, but the good news for you is that you don't need to use model.latest_version on this model to get the latest version. You can run:

prompt_input = "say hello in python"

result = replicate.run(
    "meta/meta-llama-3.1-405b-instruct",
    input={
        "top_k": 50,
        "top_p": 0.9,
        "prompt": prompt_input,
        "max_tokens": 1024,
        "min_tokens": 0,
        "temperature": 0.6,
        "presence_penalty": 0,
        "frequency_penalty": 0,
    },
)

You can find other examples of how to call this model from Python on our website.