Data Migration from Clickhouse OSS (running in EKS) to Clickhouse Cloud

[abeleon-m1]

Describe your question

Wondering if anyone's gone through the process of migrating from OSS Clickhouse running in EKS to Clickhouse Cloud. If so, how did you accomplish migrating existing data over to new cloud instance? Reading through docs, traces should be first stored in S3 before loading into Clickhouse. I see a few issues/threads on using an internal script to load data from S3.

Can I point my langfuse deployment to the new Clickhouse cloud service and load all traces/events/etc using this or another script?

Clickhouse cloud has some docs on migrating from OSS to Cloud but was wondering if there was a documented way of doing this.

Thanks in advance!

Langfuse Cloud or Self-Hosted?

Self-Hosted

If Self-Hosted

v3.100.0 OSS

If Langfuse Cloud

No response

SDK and integration versions

No response

Pre-Submission Checklist

• I have checked for existing issues/discussions and consulted Langfuse AI.

[inkeep-ai-assistant[bot]]

Based on the available documentation, here's what I can confirm about migrating ClickHouse data in Langfuse:

Configuration for ClickHouse Cloud

You can point your Langfuse deployment to ClickHouse Cloud by updating your environment variables(1):

CLICKHOUSE_URL=https://<identifier>.<region>.aws.clickhouse.cloud:8443
CLICKHOUSE_MIGRATION_URL=clickhouse://<identifier>.<region>.aws.clickhouse.cloud:9440
CLICKHOUSE_USER=default
CLICKHOUSE_PASSWORD=changeme
CLICKHOUSE_MIGRATION_SSL=true

(1)

S3/Blob Storage Architecture

The documentation confirms that Langfuse v3 uses S3/Blob Store to store all incoming events(2). The architecture stores events in S3/Blob storage before processing them into ClickHouse(2).

Data Migration Approach

While the documentation doesn't provide a specific step-by-step guide for migrating existing ClickHouse data from EKS to ClickHouse Cloud, the v2 to v3 migration guide mentions that "Information will not be deleted from the S3/Blob Store. This action needs to be performed manually by an administrator"(2), suggesting that S3 serves as the source of truth for event data.

What's Not Documented

The specific internal replay script you mentioned and the exact process for replaying S3 events into a new ClickHouse Cloud instance aren't covered in the available documentation. For this migration scenario, I'd recommend reaching out through Langfuse's support channels or opening a GitHub discussion, as this appears to be an operational procedure that may require guidance from the Langfuse team.

📚 Sources:

1. ClickHouse
2. Migrate Langfuse v2 to v3

---

Have another question? Just tag @inkeep.

[Steffen911]

Hey @abeleon-m1 ,
I would recommend to use one of the documented ways in ClickHouse as those should be significantly more performant than the script we have in the repo. This is optimized for singular backfills in case the ClickHouse nodes went down between backups and not for full history reloads.

I think pointing Langfuse to the new ClickHouse instance you'd get all new events in there and then you can try a bunch of different things to move the historic data.
I would go in the following order (from easiest to hardest):

• Use the remoteSecure function like in the link you've shared.
• Try to backup the current instance to a Cloud storage and restore that into the the Clickhouse cloud instance (https://clickhouse.com/docs/operations/backup/s3_endpoint).
• Use the S3 table function for a parquet style export and then reimport that into ClickHouse cloud (https://clickhouse.com/docs/sql-reference/table-functions/s3).

Once you've tried those, let me know what worked best for you!

[abeleon-m1]

We ended up using just doing a lift and shift using the clickhouse client. Ran into a few Langfuse Helm chart quirks, but nothing breaking. In general:

• We orphaned all clickhouse/zookeeper resources by adding the helm.sh/resource-policy: keep annotation to them.
• We can't point Langfuse to an external clickhouse db without setting the: clickhouse.deploy to false which would delete our existing in cluster db. If we tried to without setting that flag to false, we saw a PASSWORDS ERROR (since old db has different credential).
• We did need to also set CLICKHOUSE_CLUSTER_ENABLED to false. Ran into similar error as the one mentioned here. Code: 80
• Once old resources were orphaned and clickhouse.deploy set to false, Langfuse handled the schema/migrations and we simply did a clickhouse client query INSERT INTO $DEST_DB.$TABLE FORMAT JSONEachRow to backfill the data. Saw little downtime (web/worker pods rolling) and were able to backfill data with Langfuse up and receiving new traces.

Thanks for the help