Clickhouse timeout errors

[mohankn-pfizer]

hi

Observing Clickhouse query timeout errors in langfuse web container, which is making langfuse fail to load traces/langfuse UI. Clickhouse has abundant CPU and Memory, it's not hitting threshold. What could be the reason and how to fix this.

type: Selfhosted
Clickhouse CPU: 8GB, Memory: 16 GB. [30% is only utilized]

Errors on langfuse web:

[inkeep-ai-assistant[bot]]

Based on the documentation, there are several potential causes for ClickHouse timeout errors even when CPU and memory resources appear sufficient:

Performance and Query Optimization

The most common cause of slow UI queries is missing time filters(1). The tracing data is indexed by projectId and time, so adding filter conditions on those should significantly improve performance(1). If all filters are in place, a larger ClickHouse instance may increase the observed performance(1).

ClickHouse is designed to scale vertically, meaning adding more memory to the instance should yield faster response times(1). For larger deployments, at least 16 GiB of memory is recommended(1).

Data Synchronization Issues

Similar timeout issues can occur due to data mismatches between Postgres and ClickHouse(2). There might be synchronization problems where traces are available through the API but not visible in the web UI(2). This could be related to:

• Data mismatches between the data stored in Postgres and ClickHouse(2)
• Caching or synchronization issues where the Langfuse backend processes events asynchronously, introducing delays in their availability in the UI(2)

Resource Configuration

Even with abundant CPU and memory, ClickHouse may encounter memory limit errors during background operations. One user reported memory limit exceeded errors during merge operations, despite having sufficient overall resources(2).

Recommended Solutions

1. Check time filters: Ensure your UI queries include appropriate time range filters to leverage the projectId and time indexing(1)

2. Verify memory allocation: Consider increasing ClickHouse memory allocation even if current usage appears low, as ClickHouse benefits from vertical scaling(1)

3. Check data synchronization: Verify that data is properly synchronized between Postgres and ClickHouse databases(2)

4. Review ClickHouse configuration: Ensure your ClickHouse deployment has the minimum recommended 2 CPU and 8 GiB Memory(3)

If these steps don't resolve the timeout issues, the problem may require deeper investigation into your specific ClickHouse configuration and query patterns.

📚 Sources:

1. Scaling
2. Traces Disapeared on my webui, but are still available through api (python sdk)
3. Scaling

---

Have another question? Just tag @inkeep.

[jannikmaierhoefer]

Hi @mohankn-pfizer, was inkeep able to help you here?

[mohankn-pfizer]

@jannikmaierhoefer No. By default, filters are applied and Clickhouse resources are not exceeded beyond 30%. Any help / guidance would be appreciated.

[dahnny012]

where are you hosting it and how?

[mohankn-pfizer]

We are self-hosting in AWS using ECS containers.

[dahnny012]

are you using EFS as a storage solution? EFS's provisioned bursting was extremely slow for me. I swapped it to elastic and it decreased latency by 10x.

[mohankn-pfizer]

Yes, we are using EFS with elastic. intially bursting was slow for us as well. changing to elastic was working better. But recently started seeing these timeout errors in Lang fuse web.

[mohankn-pfizer]

@Steffen911 @jannikmaierhoefer Looking for some guidance on timeout errors when loading we page.

[Steffen911]

@mohankn-pfizer This also depends on the endpoints being used. In general, we have the following recommendation for APIs: https://langfuse.com/faq/all/api-524-http-errors.
For dashboards queries (the ones that you seem to have issues with) we are aware that they are slow to query over longer time-horizons and are working on more fundamental improvements. For now, limiting the time range should be the best way to address this.