Clickhouse single replica to multi replica switch

[playcan121-coder]

Describe your question

Hello, what entails moving from single replica clickhouse to 3 replica clickhouse (will use keeper) and keep the existing data, I know langfuse states its not possible and we need to make a decision before hand, but anyone in the community successfully went through this kind of activity?

Langfuse Cloud or Self-Hosted?

Self-Hosted

If Self-Hosted

3.152.0

If Langfuse Cloud

No response

SDK and integration versions

No response

Pre-Submission Checklist

• I have checked for existing issues/discussions and consulted Langfuse AI.

[inkeep-ai-assistant[bot]]

The documentation explicitly states that the number of replicas for each ClickHouse shard cannot be increased at runtime without manual intervention or downtime(1). When deploying ClickHouse on Kubernetes, a minimum of 3 replicas is recommended for production setups(1).

Based on community experience, one user successfully transitioned to a 3-replica ClickHouse cluster (1 shard, 3 replicas) but encountered issues that required specific configuration steps(2). The key requirements for a successful multi-replica setup include:

Critical Configuration Requirements:

1. Timezone Configuration: ClickHouse must be set to UTC timezone(2). One user resolved graph and trace display issues by changing ClickHouse's timezone to UTC(2).

2. Table Engine Types: All tables must use ReplicatedReplacingMergeTree or SharedReplacingMergeTree engines in cluster mode(2). Using MergeTree can cause data consistency problems or missing data in graphs and traces(2).

3. Cluster Name: If using a custom CLICKHOUSE_CLUSTER_NAME (not default), you must set LANGFUSE_AUTO_CLICKHOUSE_MIGRATION_DISABLED = true and manually run ClickHouse migrations after adjusting the cluster name in ./packages/shared/clickhouse/migrations/clustered/*.sql(1).

4. Macros Configuration: The following ClickHouse configuration parameters must be set for database migrations to work correctly(1):

<macros>
 <shard>01</shard>
 <replica>example01-01-1</replica>
</macros>

<default_replica_path>/clickhouse/tables/{database}/{table}</default_replica_path>
<default_replica_name>{replica}</default_replica_name>

(1)

Recommended Approach:

For Kubernetes deployments using the Bitnami ClickHouse Helm chart, the recommended production configuration includes(1):

clickhouse:
 shards: 1
 replicaCount: 3
 resourcesPreset: large

(1)

The migration from single to multi-replica will likely require downtime and manual intervention, as the replica count cannot be changed at runtime(1). Consider backing up your data before attempting this transition(3).

📚 Sources:

1. ClickHouse
2. bug: use clickhouse cluster on self host, the graph images or trace trace is no working, time format is failed
3. Backup Strategies

---

Have another question? Just tag @inkeep.