Traces stop appear in self-hosted Langfuse after restart pods

[vladosby]

I have self-hosted Langfuse version 3.126.1.

When I restart both the web and worker pods of Langfuse, new traces stop appearing in the UI.
The system continues running without visible errors, but incoming traces are not displayed.
After performing another restart of the same pods, traces start appearing again as expected(new traces). No any code changes happen here. No any errors in the logs.

Maybe someone has any idea how debug it?

Code that I use to send traces, does not show any errors:

from langfuse import Langfuse

langfuse = Langfuse(
    secret_key="secret",
    public_key="65cb2a151505",
    host="https://langfuse-preprod.playticorp.com",
    debug=True,
)

print("Client host:", langfuse, "env:", getattr(langfuse, "environment", "default"))

with langfuse.start_as_current_span(name="smoke-test8", input={"x": 1}) as root:
    root.update_trace(
        session_id="1",
        tags=["search-agent", "smoke"]
    )
    root.update(output={"y": 2})

langfuse.flush()

# aaa = langfuse.get_prompt(name="output_query").prompt
# print("Prompt:  " + str(aaa))

Web pod shows that trace came without any erros:
{"dd.span_id":"13226161569681507485","dd.trace_id":"15673132696946600365","langfuse.header.x-langfuse-public-key":"65cb2a151505","langfuse.header.x-langfuse-sdk-name":"python","langfuse.header.x-langfuse-sdk-version":"3.9.0","level":"debug","message":"Request to route OTel Traces projectId cmgwfra6q0005wh07nddi71nj","query":{},"span_id":"b78cc3df56ca049d","timestamp":"2025-11-06T11:51:05.162Z","trace_id":"a350407de869b4efd9822725e70fe9ad"}

The worker pod shows no errors in the logs, but it also does not perform any background processing or write data to ClickHouse, unlike in the normal working state.

[inkeep-ai-assistant[bot]]

Based on your description, this appears to be a queue processing issue in your Langfuse v3 deployment. The web container is accepting traces (as shown in your logs), but the worker container isn't processing them from the Redis queue.

Key Architecture Points

In Langfuse v3, the system uses an asynchronous architecture where the web container accepts events and queues them in Redis, then the worker container processes these events and writes them to ClickHouse(1). When traces stop appearing after a restart but reappear after another restart, this typically indicates a queue or worker connection issue.

Debugging Steps

1. Check Redis Connection and Queue Status

Verify that your worker pod can connect to Redis and that queues are being populated. The worker processes events from Redis queues named for ingestion(2).

2. Verify Worker Container Logs

Since your worker shows no errors but also no processing activity, check for:

• Redis connection establishment messages
• Queue polling activity
• Any silent connection failures

3. Check Environment Variables

Ensure these critical variables are set correctly on both web and worker pods(3):

• REDIS_CONNECTION_STRING (or alternatives)
• CLICKHOUSE_URL
• CLICKHOUSE_USER
• CLICKHOUSE_PASSWORD
• LANGFUSE_S3_EVENT_UPLOAD_BUCKET

4. Verify Queue Configuration

If you're using queue sharding, ensure LANGFUSE_INGESTION_QUEUE_SHARD_COUNT and related settings are consistent across restarts(2).

5. Check ClickHouse Connectivity

Verify the worker can connect to ClickHouse. Connection issues might cause silent failures(3).

Potential Root Causes

Based on the v3 architecture, the most likely issues are:

1. Redis connection not re-establishing properly after pod restart
2. Queue consumer not starting in the worker container
3. ClickHouse connection timing out or failing silently
4. Environment variable inconsistency between restarts

Since a second restart resolves the issue, this suggests a race condition or initialization order problem rather than a configuration error.

I recommend checking the worker pod's startup logs carefully for Redis and ClickHouse connection establishment messages, and verifying that all required environment variables are present and consistent across restarts.

📚 Sources:

1. Migrate Langfuse v2 to v3
2. Scaling
3. ClickHouse

---

Have another question? Just tag @inkeep.

[jannikmaierhoefer]

Hi @vladosby, was inkeep able to help you here?

[vladosby]

The issue was caused by a misconfiguration of Redis on our side. It was deployed as a Sentinel setup, but the configuration didn’t specify that. As a result, during Langfuse startup, it connected to the wrong Redis node (not the master).