Intermittent traces. prompts Request Timed Out issue in Langfuse

[vani760]

Describe your question

Hi Team,
we have hosted langfuse version 3 in linux VM from past 1 month we are encountering traces/prompts time out issue when the team opens the langfuse its intermittent issue , at present docker compose.yml file image versions we are using below ones, after upgrading to this vesions only we are observing timeout issue, so could please let us know the recommended version.

image: docker.io/langfuse/langfuse-worker:3.200.0
image: docker.io/langfuse/langfuse:3.200.0
image: clickhouse/clickhouse-server:26.6.1.1193
image: minio/minio:latest
image: docker.io/redis:7.4-bookworm
image: postgres:17.9-bookworm

Note: As suggested by your team i have added below 2 but still getting traces timeout issue

1. If you must stay on 26.x, set CLICKHOUSE_DISABLE_LAZY_MATERIALIZATION=true in your environment to disable the problematic optimizer [6].

2.Run OPTIMIZE TABLE traces FINAL and OPTIMIZE TABLE scores FINAL in ClickHouse to force part merges, which reduces deduplication overhead during queries

Thanks & Regards,
Vani

Langfuse Cloud or Self-Hosted?

Langfuse Cloud

If Self-Hosted

3.200.0

If Langfuse Cloud

No response

SDK and integration versions

No response

Pre-Submission Checklist

• I have checked for existing issues/discussions and consulted Langfuse AI.

[inkeep-ai-assistant[bot]]

The timeout issues you're experiencing with traces and prompts are likely related to insufficient ClickHouse resources or query performance. Here are several recommendations:

Infrastructure Sizing

The minimum recommended ClickHouse configuration is 2 CPU, 8 GiB Memory. (1) For larger deployments with slow UI queries, ClickHouse should have at least 16 GiB of memory as it's designed to scale vertically. (1)

Performance Optimizations

1. Add time filters to your queries

Tracing data is indexed by projectId and time, so adding filter conditions on those fields should significantly improve performance. (1)

2. If using OpenTelemetry exclusively, enable FINAL modifier optimization

Set LANGFUSE_SKIP_FINAL_FOR_OTEL_PROJECTS=true to skip the FINAL modifier for OTel-only projects, which reduces query-time merge work. (1)

3. Consider routing reads to separate ClickHouse compute

If using ClickHouse Cloud or BYOC, set CLICKHOUSE_READ_ONLY_URL to route UI and API read queries to a dedicated read-only compute group, isolating them from ingestion writes. (1)

ClickHouse System Tables

ClickHouse system log tables (trace_log, text_log, opentelemetry_span_log, etc.) can dominate disk usage and affect performance. You can disable unused system log tables by mounting a config file:

<clickhouse>
  <trace_log remove="1"/>
  <text_log remove="1"/>
  <opentelemetry_span_log remove="1"/>
  <asynchronous_metric_log remove="1"/>
  <metric_log remove="1"/>
  <latency_log remove="1"/>
</clickhouse>

(1)

Check Largest Tables

Run this query to identify which tables are consuming the most resources:

SELECT table, formatReadableSize(size) as size, rows FROM (
  SELECT
    table,
    database,
    sum(bytes) AS size,
    sum(rows) AS rows
  FROM system.parts
  WHERE active
  GROUP BY table, database
  ORDER BY size DESC
)

(1)

Data Retention

Consider configuring a data retention policy to automatically remove old data, which is the most effective way to free disk space and improve query performance. (1)

📚 Sources:

1. Scaling

---

Have another question? Just tag @inkeep.

[vani760]

we are using linux VM with data disc 350 GB, at present still the disc size 210 GB available in VM logs the the query is processing in sec and in code the read timeout we chnaged to 30 sec so from 2 days we not observing this issue, but what is the trace is processiong more than 30 sec again we will get timeout error what best you will suggest , anything needs to upgrade the versions? because after migrating to these versions only we are facing timeout issue before that it was working fine, 1 month back we updated these version.

Thanks
Vani

[vani760]

Hi @s,

Thank you for your suggestion.

We performed additional troubleshooting after your recommendation.

Current Environment
Langfuse Web: 3.200.0
Langfuse Worker: 3.200.0
ClickHouse: 26.6.1.1193
PostgreSQL: 17.9-bookworm
Redis: 7.4-bookworm
MinIO: latest
Deployment: Docker Compose on Azure VM
Changes already tested
Enabled:
CLICKHOUSE_DISABLE_LAZY_MATERIALIZATION=true
Executed:
OPTIMIZE TABLE traces FINAL;
OPTIMIZE TABLE scores FINAL;
Restarted the Langfuse web and worker containers.
Additional testing

Initially, our SDK was configured with a 5-second read timeout and we observed:

Failed to export span batch
HTTPSConnectionPool(...): Read timed out (read timeout=4.999...)

We then increased the SDK read timeout to 30 seconds.

However, the issue still persists and now fails after approximately 30 seconds:

Failed to export span batch
HTTPSConnectionPool(host='genai-langfuse-v3.accenture.com', port=443):
Read timed out.
(read timeout=29.999...)

This indicates that increasing the client timeout does not resolve the problem.

UI behaviour

Multiple users across DEV and QA environments intermittently observe:

Request Timed Out
Unexpected Response

The affected API paths include:

traces.filterOptions
scores.getScoreColumns
projects.environmentFilterOptions

Sometimes the request succeeds after refreshing the page, while other times it times out.

ClickHouse observations

We also checked ClickHouse.

system.query_log shows normal SELECT queries completing in milliseconds during normal operation.
No obvious long-running queries or exceptions were observed when checking after the issue.
Health endpoint is always reachable.
ClickHouse appears healthy under normal load.
Question

You suggested using a larger ClickHouse cluster.

Could you please clarify what indicators make you think the ClickHouse cluster is undersized?

Are there any specific ClickHouse metrics you recommend checking (CPU, memory, active parts, merges, background pools, disk I/O, etc.) to confirm whether this is a resource bottleneck?

Also, based on the behaviour above, do you think this issue can be resolved while staying on Langfuse v3, or would you recommend upgrading to a newer v3 release or directly to v4?

Thank you for your guidance.

[CodeCLS]

Hi @vani760, you've been creating quite a few GitHub issues over the last few days. We'd like to better understand your use case and process, so could you please contact us at support@langfuse.com with the subject line "Vani760 - GitHub Issues"? Thanks!