Self Hosted Langfuse and Clickhouse: How to increase the replica

[subhashinir]

Describe your question

We have a self hosted langfuse deployed on open shift cluster. Currently the Clickhouse is single sharded and has only one replica.
I understand that it is recommended to have 3 replicas in production and it is not possible to increase replica during runtime. If we want to increase the replica to 3 now (Clickouse already has 50GB of data in PV), what are the steps we should take?

1. Required config changes
2. Changes with respect to PV
3. How to deploy Clickhouse keeper
4. when to plan this so that the incoming traces are not lost
5. Precautions / Backup to take

Langfuse version: 3.143
Clickhouse Version: 25.8.13

Langfuse Cloud or Self-Hosted?

Self-Hosted

If Self-Hosted

3.143

If Langfuse Cloud

No response

SDK and integration versions

No response

Pre-Submission Checklist

• I have checked for existing issues/discussions and consulted Langfuse AI.

[inkeep-ai-assistant[bot]]

Increasing ClickHouse replicas from 1 to 3 in your self-hosted Langfuse deployment requires careful planning. Here's what you need to know based on the Langfuse documentation:

Required Configuration Changes

For Kubernetes deployments using the Bitnami ClickHouse Helm chart, you need to update your values.yaml(1):

clickhouse:
  deploy: true
  shards: 1 # Fixed: Langfuse does not support multi-shard clusters
  replicaCount: 3
  resourcesPreset: large # or more
  persistence:
    size: 100Gi
  auth:
    username: default
    password: changeme

(1)

Important: The number of replicas cannot be increased at runtime without manual intervention or downtime(1). Langfuse requires certain ClickHouse configuration parameters for database migrations, specifically the macros and default_replica_* configuration, which should be covered by the Helm chart without any further configuration(1).

Backup Strategy Before Changes

Before increasing replicas, implement a comprehensive backup strategy(2):

Volume Snapshots (recommended for Kubernetes):

# Create a VolumeSnapshot for each ClickHouse replica
kubectl apply -f - <<EOF
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: clickhouse-backup-$(date +%Y%m%d-%H%M%S)
  namespace: langfuse
spec:
  source:
    persistentVolumeClaimName: data-langfuse-clickhouse-0
  volumeSnapshotClassName: csi-hostpath-snapclass
EOF

(2)

Ensure you're also adding snapshots for the ClickHouse ZooKeeper volumes(2).

Persistent Volume Considerations

Your storage class must have allowVolumeExpansion = true as observability workloads tend to be very disk heavy(1). For cloud providers like AWS, GCP, and Azure, this should be the default(1).

Since you already have 50GB of data in your PV, ensure:

• Your PVCs are configured with the correct storage class
• You have sufficient disk space allocated (the documentation recommends starting with 100Gi)(1)

ClickHouse Keeper Deployment

The documentation mentions that ClickHouse Keeper (ZooKeeper) is necessary for replicated setups. When using the Bitnami Helm chart with replicaCount: 3, the chart should handle the ClickHouse Keeper deployment automatically(1).

Minimizing Trace Loss During Migration

To minimize the risk of losing incoming traces during the replica increase:

1. Plan during low-traffic periods to reduce the impact
2. Ensure backup completion before making changes(2)
3. Scale your Langfuse worker containers appropriately - the documentation recommends scaling by CPU load, with a load above 50% for a 2 CPU container indicating saturation(3)

The Langfuse worker publishes queue length metrics via statsd (langfuse.queue.ingestion.length) that can be used to monitor ingestion health(3).

Additional Precautions

• Minimum Infrastructure Requirements: Ensure your ClickHouse deployment meets the minimum requirements of 2 CPU and 8 GiB Memory(3)
• Resource Allocation: The documentation recommends at least the large resourcesPreset for production setups, and more for larger deployments(1)
• Monitor disk usage: Set up monitoring alerts for disk usage (recommend alerting at 80% capacity)(1)

Critical Note: Shards are used for horizontally scaling ClickHouse. Today, Langfuse does not support a multi-shard cluster, i.e. the shards value must be set to 1(1). The documentation advises to contact the Langfuse team if you hit scaling limits of a single shard cluster.

📚 Sources:

1. ClickHouse
2. Backup Strategies
3. Scaling

---

Have another question? Just tag @inkeep.

[subhashinir]

@inkeep Can clickhouse native backup be automated and scheduled? like backup should run weekly once?

Another question, slightly unrelated. For clickhouse can we enable hot/cold tiered storage? How to do that? We want to keep 60 days data in clickhouse PV, the older data can be moved to S3

[dahnny012]

We use ecs. Its likely easier with EKS but here was our migration strategy.

ClickHouse Cluster Migration: Overview for Engineering Review

Why Are We Doing This?

Our current Langfuse deployment runs a single ClickHouse instance on ECS Fargate. This is a single point of failure — if that container dies, restarts, or EFS has issues, all trace ingestion and dashboard queries stop. There is no replication, no failover, and no way to scale reads.

We are migrating to a 3-replica ClickHouse cluster (1 shard × 3 replicas) coordinated by a 3-node ClickHouse Keeper quorum. This gives us:

• High availability — tolerate 1 replica failure with zero downtime (NLB routes around it)
• Data durability — 3 copies of every row, replicated in real-time via ClickHouse Keeper
• Read scalability — NLB distributes queries across all 3 replicas
• Rolling maintenance — upgrade or restart replicas one at a time

Architecture

Current (Single-Node)

Langfuse web/worker
    ↓
CLICKHOUSE_URL=http://clickhouse.langfuse.local:8123
    ↓
Single Fargate Task (ClickHouse)
    ↓
EFS (shared root /)

Target (Clustered)

Langfuse web/worker
    ↓
CLICKHOUSE_URL=http://cluster.clickhouse:8123   (NLB)
CLICKHOUSE_MIGRATION_URL=clickhouse://cluster.clickhouse:9000
CLICKHOUSE_CLUSTER_ENABLED=true
    ↓
NLB (internal, cross-zone load balancing)
    ├── :8123 (HTTP) → replica-0, replica-1, replica-2
    └── :9000 (Native TCP) → replica-0, replica-1, replica-2
    ↓
┌────────────┐  ┌────────────┐  ┌────────────┐
│ CH Replica 0│  │ CH Replica 1│  │ CH Replica 2│
│  (Fargate)  │  │  (Fargate)  │  │  (Fargate)  │
└──────┬──────┘  └──────┬──────┘  └──────┬──────┘
       │                │                │
  EFS /replica-0   EFS /replica-1   EFS /replica-2
       └────────────────┼────────────────┘
                   Shared EFS filesystem
                   (isolated access points)

┌────────────┐  ┌────────────┐  ┌────────────┐
│  Keeper 0   │  │  Keeper 1   │  │  Keeper 2   │
│  (Fargate)  │  │  (Fargate)  │  │  (Fargate)  │
└──────┬──────┘  └──────┬──────┘  └──────┬──────┘
  EFS /keeper-0    EFS /keeper-1    EFS /keeper-2

Key Infrastructure Components

Component	Count	Purpose
ClickHouse Replicas	3	Data storage + query serving
ClickHouse Keepers	3	Replication coordination (Raft consensus)
NLB	1	Load balance across replicas
EFS Access Points	6	Isolated storage per replica/keeper
Route53 PHZ Record	1	cluster.clickhouse → NLB alias

What Needs to Happen for Each Stage

The migration is a 3-phase process per stage. All phases are decoupled and zero-downtime — Langfuse never stops serving traffic.

Phase 1: Deploy Cluster Infrastructure (No Langfuse Impact)

CDK config change in stack-config.ts:

DatabaseStackProps: {
    clickhouseClustered: true,
    deployClusterServices: true,
    clusterReplicaCpu: 2048,        // tune per stage
    clusterReplicaMemoryMiB: 8192,  // tune per stage
}

What happens: 3 Keeper + 3 Replica Fargate services boot alongside the existing single-node. Langfuse continues using the single-node. Zero customer impact.

Validation: Keeper quorum (imok), cluster visible in system.clusters, replication test with a dummy table.

Phase 2: Pre-Seed Schema + Backfill Data (No Langfuse Impact)

While Langfuse continues running on the single-node, we prepare the cluster:

1. Run Langfuse's clustered/*.sql migrations directly on the cluster via clickhouse-client — creates all tables with Replicated* engines and ON CLUSTER default
2. Populate schema_migrations so Langfuse recognizes all migrations as applied
3. Backfill data from single-node → cluster via INSERT INTO ... SELECT * FROM remote(...) — reads live data over the network, no Langfuse disruption
4. Verify row counts, replication status, schema_migrations parity

Why we can run migrations ourselves: Langfuse's migration SQL files are available in the Docker image (/app/packages/shared/src/server/repositories/clickhouse-migrations/clustered/*.sql) and the Langfuse GitHub repo. They are plain SQL with ON CLUSTER default and Replicated* engines. We execute them in order and mark them in schema_migrations. When Langfuse starts on the cluster, it checks schema_migrations, sees everything applied, and does nothing — same as a normal restart.

Risk mitigation: If a new Langfuse version ships additional migrations between when we pre-seed and when we flip, Langfuse will detect the delta and run only the new migrations on startup. This is safe — it's how Langfuse normally handles incremental updates.

Phase 3: Flip Langfuse to Cluster (Zero-Downtime Rolling Deploy)

CDK config change:

LangfuseStackProps: {
    useClustered: true,  // ← the flip switch
}

What happens:

1. CDK deploy triggers a rolling ECS update (minHealthyPercent: 50)
2. New Langfuse tasks start with CLICKHOUSE_URL pointing to cluster NLB
3. Old tasks drain and stop — at least 50% of tasks serve traffic throughout
4. Langfuse checks schema_migrations on startup → all applied → no migration work needed
5. Single-node continues running as rollback target

Challenges We Discovered

1. Cross-Stack CloudFormation Export Issue (Resolved ✅)

The problem: Our CDK uses two stacks — ClickhouseStack (owns ClickHouse) and LangfuseStack (owns Langfuse services). When useClustered=true, the CLICKHOUSE_URL env var switches from the single-node Cloud Map URL (a cross-stack reference) to the cluster NLB URL (a hardcoded string). CDK sees the cross-stack reference is no longer needed and tries to remove the export from ClickhouseStack. But the already-deployed LangfuseStack still imports it → CloudFormation blocks the delete.

The fix: Added a CLICKHOUSE_SINGLE_NODE_URL env var that always references the single-node, keeping the cross-stack import alive. Langfuse ignores the extra env var. This fix is already in service-base.ts and will work for all stages automatically.

Deploy order for prod: Deploy LangfuseStack first (to update the import), then ClickhouseStack.

2. Table Engine Conversion (Non-Trivial)

Langfuse maintains two separate SQL migration directories:

• clustered/*.sql → ReplicatedMergeTree + ON CLUSTER
• unclustered/*.sql → plain MergeTree

Tables created in unclustered mode cannot be converted in-place to replicated engines. ClickHouse does not support ALTER TABLE ... ENGINE = ReplicatedReplacingMergeTree. The tables must be dropped and recreated.

Our approach: Run the clustered/*.sql migrations directly on the fresh cluster ourselves, then backfill data from the old single-node via INSERT INTO ... SELECT * FROM remote(...). This avoids manual SQL conversion and eliminates the need to start/stop Langfuse for schema setup.

3. Maintenance Window Required → Eliminated (Zero-Downtime)

Previously we thought a maintenance window was needed because Langfuse had to be started against the cluster to auto-run migrations. By pre-running the migrations ourselves and backfilling while Langfuse is still on single-node, we eliminated the maintenance window entirely.

How it works:

• Backfill reads from the single-node via remote() — this is a read-only operation, no Langfuse disruption
• Some rows written during the backfill window will be on single-node only — a final catch-up remote() sync handles this
• The CDK flip is a rolling ECS deploy (minHealthyPercent: 50) — Langfuse stays available throughout
• Trace data is append-only (no updates), so eventual consistency during the brief flip is acceptable

Estimated backfill times (runs in background, no user impact):

Data Volume	Backfill Time	Notes
< 1 GB (dev/beta)	~2 min	Near-instant
10-50 GB (gamma)	~15-30 min	Run and verify
100 GB+ (prod)	30-120 min	Run catch-up sync after

4. No Official Migration Path from Langfuse

Langfuse does not provide a documented unclustered→clustered migration procedure. Our backup-and-replay approach is custom and was validated on the personal stack first.

5. EFS Performance

ClickHouse is I/O-intensive. EFS has higher latency than EBS (which we'd get with EKS/PVCs). We chose EFS because:

• Already in use for single-node
• No Kubernetes expertise on team
• EFS Elastic throughput mode handles burst workloads
• 3 replicas distribute read load

We should monitor EFS IOPS and latency after migration and evaluate if EBS (via EKS) is needed later.

POC / Testing Done

Personal Stack E2E Test (2026-04-20, account 732406385518)

Full end-to-end migration test on personal stack. Key results:

Phase 1 — Cluster Infrastructure:

• ✅ 3 Keepers formed quorum (all returned imok)
• ✅ 3 Replicas joined cluster (system.clusters shows 1 shard × 3 replicas)
• ✅ Replication functional: created ReplicatedMergeTree table, inserted rows on replica-0, verified on all 3

Phase 2 — The Flip:

• ❌ First attempt failed (cross-stack export error) → auto-rolled back, zero impact
• ✅ Applied CLICKHOUSE_SINGLE_NODE_URL fix → LangfuseStack deployed successfully
• ✅ Langfuse auto-migrated 68 schema files → created all tables with Replicated* engines
• ✅ Langfuse immediately started writing new data to cluster

Data Backfill:

• ✅ remote() connectivity: cluster can query single-node over the VPC
• ✅ Backfilled 4 tables (~980 rows) in ~2 minutes:
  • traces: 179 rows
  • observations: 502 rows
  • scores: 12 rows
  • blob_storage_file_log: 289 rows
• ✅ schema_migrations managed by Langfuse (not manually backfilled)

Post-Backfill Verification:

• ✅ All 7 replicated tables: 3/3 active replicas, queue_size=0, absolute_delay=0
• ✅ Langfuse continued writing (9 new traces appeared during backfill)

Timing:

Step	Duration
Pre-flight validation	9 min
First deploy attempt (failed)	9 min
Diagnosis + fix + redeploy	15 min
Post-flip validation	10 min
Data backfill + verification	10 min
Total	53 min
Backfill only	~2 min

What's Left to Test

• Failover: kill one replica → verify Langfuse stays up via NLB
• Recovery: bring replica back → verify replication catches up
• Load test with production-scale data volume

Rollback Plan

During Migration (Before Langfuse Resumes)

1. Revert useClustered: false in stack-config.ts
2. Deploy → Langfuse points back at single-node
3. Single-node still has all original data (never modified)

After Migration (Langfuse Running on Cluster)

1. Scale Langfuse to 0
2. Revert useClustered: false, deploy
3. Scale Langfuse back up on single-node
4. Data written after the flip is on the cluster only — use S3 event backup or remote() in reverse to recover

Nuclear Option

1. Restore single-node EFS from AWS Backup snapshot
2. Revert all CDK changes, deploy
3. Data since snapshot is lost

Rollout Plan

Stage	Phase 1 (Infra)	Phase 2 (Schema + Backfill)	Phase 3 (Flip)	Notes
Personal ()	✅ Done	✅ Done	✅ Done	Full E2E validated
Beta	Next	After P1 validation	After P2 verification	Low data volume, fast backfill
Gamma	After beta	After P1 validation	After P2 verification	Medium data, ~30 min backfill
Prod (us-east-1)	After gamma	After P1 validation	After P2 verification	Largest data, longest backfill
Prod (us-west-2)	After us-east-1	After P1 validation	After P2 verification	Same procedure
Prod (eu-south-2)	After us-west-2	After P1 validation	After P2 verification	Same procedure

All phases are zero-downtime. No maintenance window required at any stage.

Open Questions for Team Discussion

1. Prod data volume — Need to measure actual ClickHouse data size to estimate backfill time
2. Monitoring — What CloudWatch alarms/dashboards do we need for the cluster?
3. Single-node cleanup — How long to keep the single-node running after migration? (Recommended: 1-2 weeks)
4. Keeper resource sizing — Current: 512 CPU / 1024 MiB. Sufficient for prod?
5. Catch-up sync strategy — After initial backfill, how many catch-up remote() syncs before the flip? (Recommended: 1 final sync with timestamp filter, then flip immediately)

Related Documents

• Cluster Implementation Plan — CDK design decisions, file structure, ECS vs EKS rationale
• Migration Playbook — Step-by-step procedure with scripts
• E2E Test Journal — Detailed minute-by-minute test log
• Cluster Debugging Guide — Troubleshooting commands
• Scaling Replicas — How to add/remove replicas