S3 Event Bucket vs ClickHouse Storage — Is the Data Redundant?

[tushar-toddle]

S3 Event Bucket vs ClickHouse Storage — Is the Data Redundant?

Hi Langfuse team,

I’m trying to better understand the relationship between the S3 event bucket and ClickHouse storage in a self-hosted Langfuse deployment.

From what I’ve observed, for each trace/observation, the data appears to exist in both:

• The S3 event bucket
• ClickHouse tables

With the data retention policy set to 0 days (indefinite retention), it seems that the same data is being retained indefinitely in both places.

Questions

1. What is the primary purpose of the S3 event bucket once the events have been ingested into ClickHouse?

   • Is S3 intended to be a backup/durable source of truth?
   • Or is it an ingestion/event-log layer that is eventually expected to be cleaned up?
   • Does Langfuse depend on older S3 events for any runtime functionality after they have been ingested into ClickHouse?

2. Is the data in S3 and ClickHouse effectively duplicate storage?

   For example, if I have a trace that is one month old and the complete data for that trace exists in ClickHouse, is there still a reason to retain the corresponding S3 event indefinitely?

3. Can S3 be used as a backup/recovery mechanism for ClickHouse?

   If the intention is to retain data indefinitely, my assumption was that it would be more cost-effective to:

   • Keep the operational data in ClickHouse.
   • Take periodic ClickHouse backups/snapshots.
   • Retain S3 event data for only the period required for ingestion/recovery.

   Is this a valid approach for a self-hosted Langfuse deployment, or does Langfuse recommend retaining the S3 events indefinitely?

We currently want to keep our trace/observation retention indefinite, but we don't necessarily need two indefinite copies of the same data if one of them can serve as a reliable backup.

What I'm trying to understand

I'd primarily like to understand the intended data lifecycle:

Application → S3 Event → ClickHouse → ?

Specifically:

• Is S3 the source of truth?
• Is ClickHouse the source of truth after ingestion?
• Is S3 required for reprocessing/recovery?
• Can old S3 events safely be deleted after successful ingestion into ClickHouse?
• If not, what is the recommended retention period for S3 events? And can we set this up, without deleting data from clickhouse?
• What backup strategy does Langfuse recommend for a self-hosted deployment where trace retention needs to be indefinite?

If the S3 events are primarily an ingestion mechanism and are not required once the data is successfully persisted in ClickHouse, we'd prefer to retain ClickHouse indefinitely and use ClickHouse backups rather than keeping a second full copy of the events in S3.

Would appreciate clarification on the intended architecture and recommended production setup for this use case.

Langfuse Cloud or Self-Hosted?

Self-Hosted

If Self-Hosted

v3.218.0 OSS

Pre-Submission Checklist

• I have checked for existing issues/discussions and consulted Langfuse AI.

[inkeep-ai-assistant[bot]]

The blob storage bucket is used to store raw events, multi-modal inputs, batch exports, and other files (1). ClickHouse is the main OLAP storage solution within Langfuse for Trace, Observation, and Score entities (2).

On the runtime dependency question: by default the Langfuse worker reads the existing event from ClickHouse and merges it with any incoming data. For projects that were not migrated from a previous version of Langfuse, this ClickHouse read is optional because the full event history is available in S3 — you can skip it by setting LANGFUSE_SKIP_INGESTION_CLICKHOUSE_READ_MIN_PROJECT_CREATE_DATE to a date in the past before your first project was created, e.g. 2025-01-01 (1). Note that any S3/blob storage deletion lifecycle rules in combination with late updates to events may cause duplicates in the event history; with the default SDK or OpenTelemetry integrations this should not affect you (1).

On trimming S3 without touching ClickHouse: you can implement lifecycle rules to automatically remove old files from your blob storage. The recommendation is to keep events for as long as you want to access them within the UI or update them, and for most customers a default of 30 days is a good choice (1). This does not apply to the media bucket used for storing uploaded media files — setting a retention policy there is not recommended because referenced media files in traces would break and future uploads of the same file would fail since upload status is tracked by hash in Postgres (1). For media files, use the Langfuse data-retention feature instead (1).

So for your setup, a blob-storage lifecycle rule on the event bucket while leaving ClickHouse retention indefinite is the documented way to avoid the second growing copy (1).

Two related points I don't have documented answers for: an explicit "source of truth" designation between S3 and ClickHouse after ingestion, and a recommended ClickHouse backup/snapshot strategy for indefinite retention. What is documented on migration is that when moving to a different ClickHouse instance, the recommended approach is to point Langfuse at the new instance so incoming data is captured there, then create a backup of the existing instance to blob storage and restore it into the new instance (2).

For managing ClickHouse growth itself, the most effective lever is a data retention policy (1). If that isn't in your plan, ClickHouse TTL can be applied to the traces, observations, scores, and event_log tables (1). You can find the largest tables with:

SELECT table, formatReadableSize(size) as size, rows FROM (
 SELECT
 table,
 database,
 sum(bytes) AS size,
 sum(rows) AS rows
 FROM system.parts
 WHERE active
 GROUP BY table, database
 ORDER BY size DESC
)

(1)

Also worth checking: on default ClickHouse configurations the system log tables (trace_log, text_log, opentelemetry_span_log, asynchronous_metric_log, metric_log, latency_log) can dominate disk usage since they have no TTL by default, and Langfuse does not read from them (1). You can disable them via a config file in /etc/clickhouse-server/config.d/ (1):

<clickhouse>
 <trace_log remove="1"/>
 <text_log remove="1"/>
 <opentelemetry_span_log remove="1"/>
 <asynchronous_metric_log remove="1"/>
 <metric_log remove="1"/>
 <latency_log remove="1"/>
</clickhouse>

(1)

Keep query_log, part_log, and error_log enabled — they are useful for debugging and remain small (1). On the Bitnami ClickHouse Helm chart, you can ship the same block via clickhouse.extraOverrides (1).

For larger deployments where ClickHouse disk growth becomes an operational bottleneck, ClickHouse Cloud or BYOC separates storage from compute so storage can grow independently of the compute serving queries — this does not replace retention policies, but removes much of the manual disk planning and volume expansion required in self-managed OSS ClickHouse setups (1).

📚 Sources:

1. Scaling
2. ClickHouse

---

Have another question? Just tag @inkeep.

[CodeCLS]

Hi @tushar-toddle, did Inkeep's answer help?