Migration between databases

[julian-hecker]

Hi all, I am migrating Langfuse v2 to Langfuse v3. However, I will be using a different database on different infrastructure after the migration. I would like to manually run background migrations from a specific date. Is this possible?

[inkeep-ai-assistant[bot]]

Based on the documentation, you can customize background migrations to start from a specific date by modifying the migration cursor in the background_migrations table(1).

The background migrations use date-based cursors to track progress(1). You can manually adjust the starting point by editing the background_migrations table in your Postgres database and updating the cursor position(2).

For deployments created before May 2024, there's a specific issue where the migration script may get stuck because all traces have the same created_at timestamp(2). In such cases, you can modify the migration script to use the timestamp field instead of created_at as the cursor(2).

To customize the migration starting point:

1. Access your Postgres database and locate the background_migrations table(1)
2. Find the relevant migration entry (traces, observations, or scores migration)
3. Update the cursor/maxDate value to your desired starting date(2)
4. You can also adjust the batchSize by adding { "batchSize": 2500 } to the args column(1)

The migration scripts can be customized by modifying the files in /worker/src/backgroundMigrations/ to use different date fields as cursors if needed(1).

Keep in mind that background migrations are executed on startup of the worker container and run until completion(1). You can monitor their progress through the Langfuse UI by clicking on the version tag and selecting "Background Migrations"(1).

📚 Sources:

1. Background Migrations
2. bug: Migration from v2 to v3 stuck for deployments created before 202405

---

Have another question? Just tag @inkeep.

[jannikmaierhoefer]

Hi @julian-hecker, was inkeep able to help you?

[julian-hecker]

Hi @jannikmaierhoefer! Yes, I think this is what I was looking for. Thanks :)