Is running ClickHouse migrations on every web container startup safe when scaling? (golang-migrate concurrency)

[furqan-shaikh-dev]

Describe your question

I noticed that the Langfuse web container runs ClickHouse migrations at startup via: entrypoint.sh → clickhouse/scripts/up.sh → migrate up. If multiple web pods start simultaneously (e.g., a Deployment with >1 replica), all of them execute migrate up in parallel.

My questions:

1. Does Langfuse consider this concurrency safe for ClickHouse?
2. What happens if two pods attempt the same CREATE TABLE or ALTER TABLE migration at once?
3. Is there any recommended best practice for production to avoid migration races?
   Thanks!

Langfuse Cloud or Self-Hosted?

Self-Hosted

If Self-Hosted

No response

If Langfuse Cloud

No response

SDK and integration versions

No response

Pre-Submission Checklist

• I have checked for existing issues/discussions and consulted Langfuse AI.

[inkeep-ai-assistant[bot]]

Based on the available sources, running ClickHouse migrations concurrently from multiple web containers can cause issues. Here's what the documentation reveals:

Migration Concurrency Issues

When migration errors occur, the migration tool marks your database as 'dirty' to prevent additional migrations from running on an inconsistent database state(1). The sources indicate that race conditions between container startup and ClickHouse availability cause migration failures(2).

ECS task restarts can interrupt migrations mid-process, leaving them in a dirty state(2). Connection issues between the web container and ClickHouse during startup create race conditions(2).

What Happens During Concurrent Migrations

If you encounter a migration error, you'll see an error message like "Dirty database version 1. Fix and force version" even after fixing the erred migration(1). This means your database has been marked as 'dirty'(1).

Recommended Best Practices

The documentation provides several prevention strategies(2):

1. Disable auto-migrations and run manually: Set LANGFUSE_AUTO_CLICKHOUSE_MIGRATION_DISABLED=true and run migrations manually during maintenance windows(2).

2. Increase startup probe timeouts: Give the web container more time to complete migrations before considering it unhealthy(2).

3. Ensure ClickHouse readiness: Verify ClickHouse is fully available before starting web containers(2).

Recovery from Failed Migrations

If migrations fail and mark the database as dirty, you need to investigate whether the migration was partially applied or not applied at all(1). After determining the actual state, force your database to the version that accurately reflects its current state(1).

The safest approach for production environments with multiple replicas is to disable automatic migrations and run them manually during controlled maintenance windows to avoid concurrent execution issues.

📚 Sources:

1. Clickhouse handling failed migrations in self-hosted Langfuse
2. ClickHouse migrations marked as "dirty" prevent Langfuse Web from starting

---

Have another question? Just tag @inkeep.

[Steffen911]

@furqan-shaikh-dev Does this answer cover your questions?

[furqan-shaikh-dev]

Thanks. So the recommended approach is : The safest approach for production environments with multiple replicas is to disable automatic migrations and run them manually during controlled maintenance windows to avoid concurrent execution issues?
Does this also apply to Postgres?
Thanks

[Steffen911]

@furqan-shaikh-dev Usually, we see that one pod starts and marks the migration as dirty and other containers starting around the same time will fail for this reason. When the initial pod succeeds, all other new pods will then succeed in their startup with nothing to do. Therefore, I don't expect this issue to occur often. The main recommendations here are 2. (Startup Probe timeouts) and 3. (Ensure ClickHouse is ready before Langfuse is being deployed).
If you want to avoid migration issues fully, disabling the ClickHouse migrations should work well. You'll have to watch whether there were any migrations on very upgrade though.

The Postgres migrations are safer in their nature due to the locking and consistency guarantees in Postgres. I'm not aware of a high rate of issues around Postgres migrations.

[furqan-shaikh-dev]

Thanks Steffen for the prompt response. Sorry, a QQ: marks the migration as dirty , where is this done?

[Steffen911]

@furqan-shaikh-dev The migration script for clickhouse sets the migration number, e.g. 31 to dirty = 1 in the schema_migrations table in ClickHouse. Once the migration is completed, a new row with 31 and dirty = 0 is written that tells the other pods that the migration was completed successfully.