Self-Hosted with ElastiCache and RDS Aurora PostgreSQL - UI Doesn't show traces

[YeliseS]

Deployment Arch:

• Self-Hosted
• Web, Worker, ClickHouse and Zookeeper on EKS (AWS Kubernetes)
• AWS ElastiCache (cluster mode enabled) for Redis
• AWS RDS Aurora PostgreSQL

Problem:

• Langfuse UI doesn't show any traces.

Tests performed:

• Created a test program (Python), that has Authentication, creation of a trace Id, Creation of span, creation of generation, updating to generation, end generation, end span and flush: All success, no errors raised.
• AI APP: Send traces to OTEL Collector through AI Driven app. Otel Collector exports to langfuse: All success, no errors shown in debug logs.

Troubleshoot:

• Enabled debug mode for web and worker
• Verified logs for web and worker
  • web log: "Request to route Otel Traces projectID "
    "Request to route Get Traces projectID "
    verify: Confirmed the project ID matching with the id in postgre database
  • worker log: Every 5 seconds a bunch of logs indicating "ioredis: reds write command, evalsha, posthog-integration-queue, data-retention-queue, blog storage-integration-queue, dataset-run-item-upsert-queue, secondary-ingestion-queue, create-eval-queue and few more messages". No errors shown
  • ClickHouse Log: Continuos messages and no errors: system.text_log, system.trace_log, system.latency_log. All these says "Selected parts from ..."
  • ElastiCache CloudWatch log: no new logs written

What am I missing? Any insights would be greatly appreciated!

[inkeep-ai-assistant[bot]]

Based on your description, this appears to be a Langfuse v3 deployment issue where traces are being ingested but not appearing in the UI. The logs you're seeing suggest the system is functioning, but there may be a configuration or processing issue.

From the available documentation, here are the key areas to investigate:

Redis Configuration Issue
Your setup uses AWS ElastiCache with cluster mode enabled. The documentation indicates potential Redis compatibility issues (1). You should verify that your Redis connection string is properly configured for cluster mode, as Langfuse requires specific Redis configurations.

Background Migration Status
Since you're running Langfuse v3, background migrations are critical for data visibility (2). The traces may be queued but not yet migrated to ClickHouse. Check your background migration status in the Langfuse UI admin panel or logs to ensure the following migrations have completed (2):

• Traces migration from Postgres to ClickHouse
• Observations migration
• Scores migration

Asynchronous Processing
Langfuse v3 uses asynchronous processing where events are queued and processed by the worker container (2). The worker logs showing queue activity every 5 seconds suggest the queuing system is working, but you should verify that events are actually being processed from the queues and written to ClickHouse.

Environment Variables
Ensure all required v3 environment variables are properly set (2):

• CLICKHOUSE_URL
• CLICKHOUSE_USER
• CLICKHOUSE_PASSWORD
• REDIS_CONNECTION_STRING
• LANGFUSE_S3_EVENT_UPLOAD_BUCKET

The fact that you're seeing "Request to route Get Traces" in the web logs but no traces in the UI suggests the data pipeline between your ingestion, worker processing, and ClickHouse storage may have an issue.

I'd recommend checking the GitHub Discussions for similar ElastiCache and RDS Aurora issues, as this appears to be a deployment-specific configuration problem (1).

📚 Sources:

1. Troubleshooting & FAQ
2. Migrate Langfuse v2 to v3

---

Have another question? Just tag @inkeep.

[YeliseS]

What are some of the key string patterns from logs would help me confirm:

• Web connected to Redis to push message to ElastiCache (Redis) Queue: Success vs Failure to connect
• Web connected to S3 bucket to push the raw payload received: I veified Event Buck has data
• Webworker connected to ElastiCache (Redis) Queue?
• Migration between Redis and Clickhouse is success vs failure

Appreciate any insights.

[dahnny012]

check the queue in elastic cache via the admin api.

[YeliseS]

The queues exist but are empty... will verify few more things little later today as my privilege (IAM) limiting my validation

[dahnny012]

Just ran into a similar issue since my client was hosted in lambda. The lambda didn't have a chance to flush when the request ended. I needed to either flush manually or try to set LANGFUSE_FLUSH_AT: 1

[YeliseS]

Thank you for your response. I added this env variable to both Web and Worker then redeployed but no difference in behavior or logs.

[Steffen911]

@Sai4Github Can you follow our troubleshooting guide and share your findings?

[YeliseS]

@Steffen911 BullMQ keys are piling up and "bull:ingestion-queue: delayed count increasing. Tried reprocess but no luck they are going back to Delayed. It seems no consumers.
Web and Worker logs are clean, no indication of any errors/failures. Web log only indicates request route to correct projectId.
S3 bucket content are increasing.
Switched from Cluster ElastiCache (Cluster Mode Enabled ) to Single-Non non cluster, but no luck. This changed Egine from ReplicatedMergeTree to ReplacingMergeTree.
ClickHouse tables are empty.
Are there any specific Environment Variable(s) I might be missing ?

[YeliseS]

@Steffen911 Appreciate your response. Sorry I could't find any document but this link of FAQs
https://langfuse.com/self-hosting/troubleshooting-and-faq

These are too generic I felt. Following are only the article I felt relevant:

https://langfuse.com/faq/all/self-hosting-missing-events-after-ingestion
https://langfuse.com/faq/all/self-hosting-queue-management-bullmq-admin-api

I followed these, these are more of verification steps but not suggesting any actions,
how to figure why ClickHouse tables are not getting populated and what action to perform?
Why BullMQ might go in Delayed. What specific fields in the payload are missing etc.

Please take this as a feedback with a pinch of salt if I am missing the details.

Update:
I have upgraded to from V3.108.0 to V3.121.0. Seems the migration was smooth.
However the web server shows "Trace created not found with authorized project"
AFter upgrade I created new Organization, project and API Keys. I used the same. This is when I try to create the Trace ID directly from a client program.
I verified PostgreSQL tables Organizations, Projects and Api_keys tables and confirm the keys existence and their relation.
If I post my client api trace through Otel Collector then in the langfuse web log I see "Request to route Otel Traces projectID "; "Request to route Create Trace projectID "; "Request to route Create Span projectID "

But UI not getting populated with any data. ClickHouse tables "traces", "observations" are empty.

[YeliseS]

Update - Issue resolved. Worker was configured with BullMQ-Prefix but not web.

[Steffen911]

@Sai4Github Can you share the difference in the configuration before and after? This is the first time I hear about this problem and it could be useful to include it in the troubleshooting guide.

[YeliseS]

Below node option to force a custom prefix for BullMQ is set for WORKER but the same was missing for WEB.
NODE OPTIONS: "--require /config/force-bull-prefix.js"

Because of this mis match, Jobs created by WEB are invisible for WORKER.