Migrating between MongoDB and PostgreSQL

Migrating between MongoDB and PostgreSQL

This page covers the built-in one-time remote database migration flow in ytdl-material.

Use this when you want to move:

• MongoDB -> PostgreSQL
• PostgreSQL -> MongoDB

If you are moving from the local JSON DB to a remote DB, do not use ytdl_db_migrate. See Configuration and Environment Variables instead.

What the migration does

On startup, ytdl-material can copy all of its data from one remote database to the other before the app starts normally.

• ytdl_db_migrate=postgres means MongoDB -> PostgreSQL
• ytdl_db_migrate=mongo means PostgreSQL -> MongoDB

The source DB is left untouched if the migration fails.

Requirements

Before starting a remote-to-remote migration, all of the following must be true:

• ytdl_use_local_db must be false
• ytdl_mongodb_connection_string must be set
• ytdl_postgresdb_connection_string must be set
• ytdl_db_migrate must be set to either postgres or mongo
• the target database must be empty

If the target database already contains data, startup will fail instead of merging or overwriting existing records.

MongoDB -> PostgreSQL

Set:

ytdl_use_local_db: 'false'
ytdl_mongodb_connection_string: 'mongodb://ytdl-mongo-db:27017'
ytdl_postgresdb_connection_string: 'postgresql://ytdl-material:ytdl-material@ytdl-postgres-db:5432/ytdl-material'
ytdl_db_migrate: 'postgres'

Then start the app.

On a successful startup migration:

• data is copied from MongoDB to PostgreSQL
• PostgreSQL becomes the saved remote DB type
• the saved ytdl_db_migrate setting is cleared automatically

After the migration succeeds, remove the ytdl_db_migrate environment variable from your container/compose config so it is not applied again on the next boot.

PostgreSQL -> MongoDB

Set:

ytdl_use_local_db: 'false'
ytdl_mongodb_connection_string: 'mongodb://ytdl-mongo-db:27017'
ytdl_postgresdb_connection_string: 'postgresql://ytdl-material:ytdl-material@ytdl-postgres-db:5432/ytdl-material'
ytdl_db_migrate: 'mongo'

Then start the app.

On a successful startup migration:

• data is copied from PostgreSQL to MongoDB
• MongoDB becomes the saved remote DB type
• the saved ytdl_db_migrate setting is cleared automatically

After the migration succeeds, remove the ytdl_db_migrate environment variable from your container/compose config so it is not applied again on the next boot.

Important behavior

The target DB must be empty

This migration is intentionally strict.

• it will not merge data from both databases
• it will not append into a populated target
• it will abort startup if the target already has data

Failed migration retries

If startup fails and you leave ytdl_db_migrate in the environment, the container will try the migration again on every restart.

That is expected. Remove or fix the migration env vars before restarting if you do not want another migration attempt.

Local JSON DB is different

For local JSON -> remote DB, set ytdl_use_local_db=false and provide the remote connection string.

If the configured remote DB is empty and the local DB has data, ytdl-material will automatically copy the local DB into that remote DB on startup.

Do not set ytdl_db_migrate for local JSON -> remote moves.

If a previous PostgreSQL attempt failed

If you previously tried a migration with an older build and the PostgreSQL target was already created, it may be safest to recreate the target PostgreSQL database or schema before retrying.

For example:

DROP SCHEMA public CASCADE;
CREATE SCHEMA public;

Or remove and recreate the PostgreSQL volume/container if that is how you manage it.

Do not wipe the source database unless you have verified the migration succeeded and the app is running against the new target.