rate limit issue when calling evalution in aws self hosted environment.

[stilist-dh]

Our langfuse system was built in aws self hosted not cloud.

When i using evlaution function(LLM as a judge) and executing evaluator, I found the problem that not execute my custom evalutor.

The status of evalutor is still pending.

So I thought the problem was the LLM calling outside the aws environment, but I checked network to outside is good.

And then I checked the log. (aws ecs that service langfuse ) and found the 'rate limited' error log.

I thought the rate limit is applied to cloud envrionment, but our envrionment is self hosted in aws.

Why this error occur and how to fix this problem????

[dosubot[bot]]

The "rate limited" error in your self-hosted Langfuse environment happens when the external LLM provider (like OpenAI, Anthropic, Bedrock, etc.) returns an HTTP 429 (Too Many Requests) response. Langfuse does not impose its own rate limits—this behavior is the same in both cloud and self-hosted modes. When a rate limit is hit, Langfuse's worker will automatically retry the evaluation job after a delay (e.g., 48 seconds), and will keep retrying for up to 24 hours before giving up. This is why your evaluator stays in "pending" status if the LLM provider keeps rate limiting your requests see code details.

To resolve this, you can reduce the frequency of evaluation jobs, upgrade your LLM provider plan to get higher rate limits, or rotate/add more API keys if your provider supports it. There are no configuration options in Langfuse to disable or bypass this retry logic for self-hosted users see retry logic. Monitoring your LLM provider's usage and error logs can help you spot and address rate limiting more proactively.

To reply, just mention @dosu.

---

Docs are dead. Just use Dosu.

[stilist-dh]

@dosubot
I already checked rate limit of LLM Provider. This is not related to LLM limit.

[dosubot[bot]]

Langfuse only triggers the "rate limited" retry logic if it receives a 429 or 5xx error from the LLM call—this can come from anywhere in the request chain, not just the LLM provider itself. If you've confirmed the LLM provider is not rate limiting, it's likely that something else (like a proxy, API gateway, custom adapter, or even a misconfigured integration) is returning a 429 error to Langfuse's evaluation worker (source).

I recommend checking for any proxies or middleware between Langfuse and your LLM provider, reviewing the logs of your evaluation worker and any custom LLM adapters for error details, and enabling debug logging to capture the full error trace. Also, try running a direct request (e.g., curl or SDK call) from your ECS container to the LLM endpoint to confirm you don't get a 429 outside of Langfuse. If you can share more detailed logs or error messages, I can help interpret them.

To reply, just mention @dosu.

---

Share context across your team and agents. Try Dosu.

[tencyle-guyg]

I'm having the same issue.
Although my llm configuration is good(used by other services as well,work fine there),
i keep getting this rate limit error on all my evaluation jobs,they are all stuck.

[stilist-dh]

Up to now, I found that when i calling evalutor using LLM-as-a-judge, there is so many requests.
it shows in aws ec2 > load balancer monitoring.
The moment the evaluator runs, request suddenly increase.
I think because of this increasing, rate limit error occured.

I executed only 5 traces filtered, why the request over 100 requests?

Any bug in langfuse? or Any options to control this?

[pander-dancer]

Hi there, I actually encountered a similar issue not long ago. After some trial and error, I found that adjusting the prompt to explicitly specify outputting JSON-formatted key-value pairs for the "score" and "reasoning" fields worked effectively. Specifically, by reinforcing the requirement for strict JSON structure with those two fields, the model responded as intended. If you're facing the same challenge, this approach might be worth a try—though your mileage may vary depending on the specific context, of course. Happy to share more details if needed!

[tencyle-guyg]

After lots of debugging and going deep into the code,i found that the aws WAF which wraps the alb pointing to my app was blocking the requests as the payload(contains the whole system+user prompt) was to big.after adjusting the WAF,it started working :)

[jannikmaierhoefer]

Hi @stilist-dh. Thanks for the update on this. I'm glad you found the issue!