S3 / Object Storage migration approach recommendations

[andresC98]

Describe your question

Do you recommend any specific approach when migrating to a different s3 bucket (or object storage system) when having Langfuse self hosted? In the event we wanted to migrate our object storage / bucket to a different one; is there any approach to follow in order not to lose existing data and make the migration seamless to users (w/out data loss)?

Langfuse Cloud or Self-Hosted?

Self-Hosted

If Self-Hosted

3.149.0

If Langfuse Cloud

No response

SDK and integration versions

No response

Pre-Submission Checklist

• I have checked for existing issues/discussions and consulted Langfuse AI.

[inkeep-ai-assistant[bot]]

The official documentation doesn't provide a specific migration procedure for moving between S3 buckets or object storage systems. However, based on the available information, here's what you need to know:

Langfuse uses S3/blob storage to store raw events, multi-modal inputs, batch exports, and other files(1). The system allows you to configure separate buckets for different use cases using environment variables(1):

• LANGFUSE_S3_EVENT_UPLOAD_BUCKET for raw event information (mandatory)(1)
• LANGFUSE_S3_MEDIA_UPLOAD_BUCKET for multi-modal tracing (optional)(1)
• LANGFUSE_S3_BATCH_EXPORT_BUCKET for batch exports (optional)(1)

Each configuration supports a prefix parameter (ending with /) to store data within a subpath of the bucket(1).

Recommended approach for migration:

Since there's no documented migration procedure, you would need to:

1. Copy all existing data from your current bucket(s) to the new bucket(s) using your cloud provider's tools or standard S3 sync utilities
2. Update the relevant environment variables to point to the new bucket(s)(1)
3. Restart your Langfuse containers with the new configuration

Important considerations:

For media uploads specifically, setting a retention policy on the media bucket is not recommended because referenced media files in traces would break and future uploads of the same file would fail since file upload status is tracked by hash in Postgres(2). This means you need to ensure all media files are successfully copied during migration.

For a zero-downtime approach similar to the v2 to v3 upgrade strategy, you could provision new Langfuse instances with the new storage configuration and shift traffic after validation(3), though this would require maintaining both storage systems temporarily.

I'd recommend reaching out to Langfuse support for specific guidance on your migration scenario to ensure no data loss.

📚 Sources:

1. S3 / Blob Storage
2. Scaling
3. Migrate Langfuse v2 to v3

---

Have another question? Just tag @inkeep.

[andresC98]

@inkeep the issue I see with that approach is that will references for s3 paths across e.g. clickhouse data be maintained after the migration? It would be nice to have some sort of "migration script" that allows backing up s3 data and do also reference management of those paths at clickhouse level.

you could provision new Langfuse instances with the new storage configuration and shift traffic after validation(3), though this would require maintaining both storage systems temporarily.

this would cause data loss though as the new instance will not have data from the old one.

[inkeep-ai-assistant[bot]]

You're correct to be concerned about S3 path references in ClickHouse. The documentation doesn't provide a built-in migration script for S3 bucket changes that automatically updates ClickHouse references(1)(2).

How S3 references work in Langfuse:

When events are stored in S3, the worker processes them and upserts data into ClickHouse(3). Once the worker has processed events from S3 into ClickHouse, the blob storage items are technically no longer needed for normal operation(3). However, the documentation recommends keeping events for as long as you want to access them within the UI or update them, with 30 days being a typical retention period(1)(3).

The reference challenge:

There's no documented automatic mechanism to update S3 path references stored in ClickHouse when you migrate buckets(1)(2). If ClickHouse contains hardcoded S3 paths pointing to your old bucket, simply copying files to a new bucket and updating environment variables won't automatically update those existing references.

Available recovery mechanism:

Langfuse does include a manual replay script for reprocessing S3 events into ClickHouse(4). This script is located at worker/src/scripts/replayIngestionEvents/ and works by:

1. Generating a CSV of S3 PUT object events using S3 access logs (e.g., via Athena)(4)
2. Placing the CSV as events.csv in the worker directory(4)
3. Running pnpm run --filter=worker refill-ingestion-events to batch events into the ingestion queue(4)

This would allow you to reprocess events from your new bucket location, but it's designed as a recovery tool rather than a migration script(4).

Recommendation:

The documentation doesn't provide a dedicated S3 migration tool that handles reference updates. For your use case, you would likely need to:

1. Copy all S3 data to the new bucket
2. Update environment variables to point to the new bucket
3. Potentially use the replay script to reprocess events if path references are broken

I recommend contacting Langfuse support directly about your specific migration scenario, as there's no officially documented migration procedure for S3 bucket changes that preserves ClickHouse references(1)(2).

📚 Sources:

1. Scaling
2. ClickHouse
3. Need help understanding the role of blob storage in Langfuse
4. ClickHouse data loss on container restart with ephemeral volumes - no S3 backfill mechanism

---

Have another question? Just tag @inkeep.

[Steffen911]

@andresC98 Do you use batch export and media uploads? Or is this only focused on event data?
Do you know what usual event update timelines for your traces are? E.g. do writes for the same observation and traces happen over multiple minutes or are they usually very short?

You can check this by looking into the upload timestamps for a few events and find the delta between the first and the last writes.

[andresC98]

Mostly event traces and some media uploads associated with multimodal traces. Regarding timelines, usually writes ocurr at the same time.

[Steffen911]

Is there a way that you can achieve near real-time replication from the old storage to the new one?