Slow loading on UI + Clickhouse Timeout Errors

[rjayaram-godaddy]

Describe your question

I have self hosted langfuse on AWS and I frequently see slower loading times and backend service overloaded errors on UI
However resource saturation does not seem to be the issue
My current configuration is:
Langfuse web( 8cpus, 16gib memory)- ECS/fargate
Langfuse worker(8 cpus, 16gib memory)- ECS/fargate
Clickhouse(16cpus, 120gib memory)- ECS/fargate with EFS(elastic mode) volumes and uses S3 buckets as default disk

Whenever I see slower loading times on UI, cpu memory metrics are very underutilized but I can see thes elogs on langfuse web and clickhouse
Langfuse web logs

2025-12-03T06:11:09.617Z error middleware intercepted error with code INTERNAL_SERVER_ERROR Timeout error. | agent-obs-langfuse
-- | --
3 December 2025, 11:41 | Error: Timeout error.
3 December 2025, 11:41 | at Socket.u (/app/web/.next/server/chunks/4374.js:73:115112) 
3 December 2025, 11:41 | at Object.onceWrapper (node:events:622:28) 
3 December 2025, 11:41 | at Socket.emit (node:events:520:35) 
3 December 2025, 11:41 | at Socket._onTimeout (node:net:604:8)
3 December 2025, 11:41 | at listOnTimeout (node:internal/timers:605:17) 
3 December 2025, 11:41 | at process.processTimers (node:internal/timers:541:7)
3 December 2025, 11:41 | 2025-12-03T06:11:09.617Z error tRPC route failed on scores.all: Database resource limit exceeded. Please use more specific filters or a shorter time range. We are continuously improving our API performance. Database resource limit exceeded. Please use more specific filters or a shorter time range. We are continuously improving our API performance.
3 December 2025, 11:41 | TRPCError: Database resource limit exceeded. Please use more specific filters or a shorter time range. We are continuously improving our API performance

Clickhouse logs

2025.12.03 06:11:22.127555 [ 23043 ] {6c14bfef-f609-43e9-8002-bc71619fb043} <Error> DynamicQueryHandler: Code: 210. DB::NetException: I/O error: Broken pipe, while writing to socket (100.100.192.243:8123 -> 100.100.192.205:54008). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below): | clickhouse
-- | --
3 December 2025, 11:41 | 2025.12.03 06:11:22.127646 [ 23043 ] {6c14bfef-f609-43e9-8002-bc71619fb043} <Error> bool DB::WriteBufferFromHTTPServerResponse::cancelWithException(HTTPServerRequest &, int, const std::string &, WriteBuffer *): Failed to send exception to the response write buffer.: Code: 236. DB::Exception: Write buffer has been canceled. (ABORTED), Stack trace (when copying this message, always include the lines below): | clickhouse
3 December 2025, 11:41 | 2025.12.03 06:11:05.838015 [ 23037 ] {cd942f60-c29e-49ac-84f0-db0e16b75eee} <Error> bool DB::WriteBufferFromHTTPServerResponse::cancelWithException(HTTPServerRequest &, int, const std::string &, WriteBuffer *): Failed to send exception to the response write buffer.: Code: 236. DB::Exception: Write buffer has been canceled. (ABORTED), Stack trace (when copying this message, always include the lines below):

I even set the KEEP_ALIVE_TIMEOUT for langfuse web and worker container higher than the ALB idle timeout. Still facing the same issue.

Is the issue related to query optimization? Or can we make some configuration changes in our self hosting environment to fix this?

Langfuse Cloud or Self-Hosted?

Self-Hosted

If Self-Hosted

3.116.1

If Langfuse Cloud

No response

SDK and integration versions

No response

Pre-Submission Checklist

• I have checked for existing issues/discussions and consulted Langfuse AI.

[inkeep-ai-assistant[bot]]

The slow UI queries and ClickHouse timeout errors you're experiencing are likely related to query optimization and ClickHouse configuration rather than resource saturation(1).

Key Issues and Solutions

1. Query Optimization

The error message "Database resource limit exceeded. Please use more specific filters or a shorter time range" indicates that your queries are too broad(1). When tracing data is queried, it's indexed by projectId and time, so adding filter conditions on those should significantly improve performance(1).

2. ClickHouse Memory Configuration

Your ClickHouse instance has 120 GiB memory, which should be sufficient, but you may need to verify the configuration. ClickHouse is designed to scale vertically—adding more memory to the instance should yield faster response times(1). The documentation recommends at least 16 GiB of memory for larger deployments(1).

3. ClickHouse Version Issue

Based on a recent GitHub issue, there's a known bug in ClickHouse version 24.3.2.23 that causes similar timeout and query parameter errors(2). The issue was resolved by upgrading ClickHouse from 24.3.2.23 to 24.8.5.115(2). Langfuse supports ClickHouse versions >= 24.3(3).

4. EFS Performance Considerations

You're using EFS (elastic mode) volumes with ClickHouse. For better performance, consider using blob storage (S3) as the primary disk for ClickHouse, as it's optimized for auto-scaling storage and increases availability and durability(3).

Recommended Actions

1. Upgrade ClickHouse: If you're running version 24.3.2.23, upgrade to at least 24.8.5.115 to resolve the known bug(2)
2. Apply time filters: Ensure all dashboard queries include specific time ranges to improve performance(1)
3. Review disk configuration: Consider configuring S3 as the primary disk for ClickHouse rather than relying solely on EFS(3)
4. Monitor ClickHouse system tables: Check if system.trace_log and system.text_log tables are growing large, as they can impact performance even on smaller deployments(1)

The "Broken pipe" errors in ClickHouse logs suggest that connections are timing out before queries complete, which aligns with the query optimization issue mentioned above(1).

📚 Sources:

1. Scaling
2. bug: Query parameter 'fillFromDate' was not set in dashboard.executeQuery (v3.120.0)
3. ClickHouse

---

Have another question? Just tag @inkeep.

[rjayaram-godaddy]

@inkeep

1. the default disk for actual data is S3 only. EFS is for persistent volumes so that data is not lost between container restarts
2. Clickhouse version is 25.9.3.48
3. In an earlier thread I was told that the trace_log,text_log system tables should not affect read/write performance of other tables

Anything else I can look into?

[inkeep-ai-assistant[bot]]

Based on your clarifications, here are additional areas to investigate:

1. ClickHouse Cluster Configuration

Since you're running ClickHouse on ECS Fargate, verify your CLICKHOUSE_CLUSTER_ENABLED setting(1). For single-container setups, this should be set to false(1). If it's incorrectly set to true, it could cause performance issues.

2. Query-Specific Performance Issues

The "broken pipe" errors suggest queries are timing out before completion(2). A similar case in the GitHub discussions showed that even with adequate resources, queries were taking 10-15 seconds due to inefficient query patterns(3). The user found that queries filtering on trace_id were particularly slow because trace_id is not in the primary key(3).

The ClickHouse table structure uses this primary key ordering(3):

PRIMARY KEY (project_id, type, toDate(start_time))
ORDER BY (project_id, type, toDate(start_time), id)

This means filtering by trace_id requires scanning many rows even with bloom filters(3).

3. Network and Connection Settings

The "Broken pipe" errors in your ClickHouse logs suggest network-level connection issues(3). Consider:

• Increasing CLICKHOUSE_MIGRATION_URL and CLICKHOUSE_URL connection timeouts
• Verifying network connectivity between your Langfuse web container and ClickHouse is stable
• Checking if AWS security groups or network ACLs are causing connection drops

4. Disk I/O Performance

While you clarified that S3 is your default disk, a similar discussion noted that EFS performance modes matter significantly(3). Even with elastic mode, throughput and IOPS limits can bottleneck ClickHouse operations(3). The discussion showed that switching from EFS bursting mode to elastic or provisioned mode improved latency from 150-300ms to 10-50ms(3).

5. Data Volume and Partitioning

The GitHub discussion showed that performance degraded exponentially after just a few thousand traces(3). With ClickHouse partitioned by toYYYYMM(start_time)(3), ensure your queries are taking advantage of partition pruning by always including time range filters.

6. ClickHouse System Resource Limits

Check if ClickHouse itself has internal resource limits that are being hit. You can query ClickHouse to identify slow queries(3):

SELECT table, formatReadableSize(size) as size, rows FROM (
 SELECT
 table,
 database,
 sum(bytes) AS size,
 sum(rows) AS rows
 FROM system.parts
 WHERE active
 GROUP BY table, database
 ORDER BY size DESC
)

This helps identify if specific tables are growing too large(2).

Given your configuration is already quite robust (16 CPUs, 120 GiB for ClickHouse), the issue is most likely query-pattern related or network connectivity between services rather than raw resource capacity.

**

📚 Sources:

1. ClickHouse
2. Scaling
3. Really slow latency for viewing traces within a timestamp and viewing a specific trace.

---

Have another question? Just tag @inkeep.

[rjayaram-godaddy]

Hey langfuse team, can anyone help me with some guidance on this? I am parallelly trying to separate the ingestion and web workload as mentioned in your scaling docsbetween 2 separate containers not sure if it will solve this problem though

[Steffen911]

@rjayaram-godaddy Are there specific queries that you frequently see timeouting? E.g. on the dashboard or the trace/observations table pages? Or is that a general behaviour across the platform?

Can you share rough metrics on the number of events within the traces, observations, and scores tables and the timeframes that you want to select data for?

[rjayaram-godaddy]

Hi @Steffen911
I could observe slow loading times in the Home page for time durations >=1 Day. For 30 min, 1 hour duration loading time is still better but not the best
It seems to occur more frequently in the sessions page. We get internal error backend service overloaded on UI
path: sessions.byIdwithScores and the debug logs in web show clickhouse timeout error during this time
Clickhouse shows the network broken pipe exception log

One more observation, these slow loading times are happening in the project which receives large number of traces/observations
To answer your question on number of events:
We ingest over 1 lakh traces/ 4 lakh observations in a day in this project.

Each trace is also associated with a session

Please see screenshots of the homepage dashboard so that you get a better idea:
Past 7 days:

Past 1 day:

One more small observation. We have another project in the same environment that receives lesser traces/observations. UI performance is much better in that project

[Steffen911]

@rjayaram-godaddy Are you using the OpenTelemetry-based SDKs for inserting trace data? In https://github.com/langfuse/langfuse/releases/tag/v3.136.0 we included a bunch of improvements around loading times if this is the case.

Could you also execute the following query for me directly on ClickHouse (replacing and <MIN_TIMESTAMP>)?

        WITH 
        deduplicated_traces AS (
          SELECT * EXCEPT input, output, metadata
          FROM traces t
          WHERE t.session_id IS NOT NULL
            AND t.project_id = <PROJECTID>
             AND timestamp >= <MIN_TIMESTAMP>
            LIMIT 1 BY id, project_id
        ),
        deduplicated_observations AS (
            SELECT *
            FROM observations o
            WHERE o.project_id = <PROJECTID>
            AND o.start_time >= <MIN_TIMESTAMP> - INTERVAL 1 HOUR
            AND o.trace_id IN (
              SELECT id
              FROM deduplicated_traces
            )
            ORDER BY event_ts DESC
            LIMIT 1 BY id, project_id
        ),
        observations_agg AS (
          SELECT o.trace_id,
                count(*) as obs_count,
                min(o.start_time) as min_start_time,
                max(o.end_time) as max_end_time,
                sumMap(usage_details) as sum_usage_details,
                sumMap(cost_details) as sum_cost_details,
                anyLast(project_id) as project_id
          FROM deduplicated_observations o
          WHERE o.project_id = <PROJECTID>
          AND o.start_time >= <MIN_TIMESTAMP> - INTERVAL 1 HOUR
          GROUP BY o.trace_id
        ),
        session_data AS (
            SELECT
                t.session_id as session_id,
                anyLast(t.project_id) as project_id,
                max(t.timestamp) as max_timestamp,
                min(t.timestamp) as min_timestamp,
                groupArray(t.id) AS trace_ids,
                groupUniqArray(t.user_id) AS user_ids,
                count(*) as trace_count,
                groupUniqArrayArray(t.tags) as trace_tags,
                anyLast(t.environment) as trace_environment
                -- Aggregate observations data at session level
            FROM deduplicated_traces t
            WHERE t.session_id IS NOT NULL
                AND t.project_id = <PROJECTID>
                 AND timestamp >= <MIN_TIMESTAMP>
            GROUP BY t.session_id
        )
        SELECT 
          session_id,
          max_timestamp,
          min_timestamp,
          trace_ids,
          user_ids,
          trace_count,
          trace_tags,
          trace_environment
        FROM session_data s
        WHERE s.project_id = <PROJECTID> AND min_timestamp >= <MIN_TIMESTAMP>
        ORDER BY min_timestamp DESC
        LIMIT 50 OFFSET 0

This is the query that initially fills the sessions table. By running it directly on ClickHouse and comparing that to your UI loading time, we can check if the problem is more within the Langfuse application stack or within ClickHouse.

I suspect that a problem might be in your single-instance setup as this will have a single node performing a lot of S3 operations which are eventually throttled based on the network allowance. Spreading the workload to multiple nodes within the cluster usually has a significant, positive effect on throughput.

[rjayaram-godaddy]

@Steffen911 I am using the api/public/ingestion endpoint to ingest traces
Let me try running your query on clickhouse container cand get back to you

[Steffen911]

Changing to the new opentelemetry based endpoint should have a significant, positive impact on the throughput and your query performance.

[rjayaram-godaddy]

@Steffen911 Any advice on the above?

[rjayaram-godaddy]

@Steffen911 Some more context on this
I was able to replicate the slow UI loading+ timeout errors on clickhouse on another environment
It seems that when we enter a timerange with large number of datapoints, and within that timerange traces/observations are also getting ingested continously in realtime this issue seems to come.

Will separating ingestion and web into 2 containers solve this problem? Or does this have to be solved at clickhouse level in terms of query optimization?

[Steffen911]

@rjayaram-godaddy That depends on the results that you see on the ClickHouse query. With the current data I cannot judge whether this comes from the web containers or the clickhouse instances.
Do you see any spikes in CPU consumption on the web container while this happens?

[rjayaram-godaddy]

@Steffen911
In the high throughput environment when I ran the query given by you in the previous thread it took a lot of time.( around 60 seconds)

this was the output I received

50 rows in set. Elapsed: 61.042 sec. Processed 104.01 thousand rows, 20.17 MB (1.70 thousand rows/s., 330.50 KB/s.)
Peak memory usage: 51.44 MiB.

Also in the environment where I simulated this issue, I increased the load of ingestion and now the UI becomes completely unresponsive. It just keeps loading indefinitely. Web and worker give timeout errors

The cpu utilization for clickhouse ecs fargate has jumped to 46% now. Web and worker dont show any spikes
During the high load I could see an unresponsive UI and occasionally got this error message

Could you please suggest the way forward now?

[Steffen911]

@rjayaram-godaddy Given those metrics, it seems that the bottleneck is rather on the ClickHouse read side. You may also benefit from splitting the web container into one for ingestion and a regular one.
I would focus on patching your ClickHouse setup though to support higher scale.