Configuration

Configuration

DBMigrator uses a single YAML file (default: config.yaml). Passwords and secrets should be referenced via environment variables.

Environment Variable Interpolation

Any config value can reference an environment variable with ${VAR_NAME}. The process exits with an error if the variable is not set at runtime.

password: "${SOURCE_DB_PASSWORD}"

YAML comment lines (starting with #) are never interpolated — ${...} inside a comment is ignored even if the variable is not set. This means documentation comments in your config file won't cause startup failures.

Full Schema

migration:
  source:
    host: "string"          # required
    port: 5432              # required
    database: "string"      # required
    user: "string"          # required
    password: "string"      # required (use ${ENV_VAR})
    schema: "public"        # optional, default: public
    ssl_mode: "Prefer"      # optional: Disable | Allow | Prefer | Require | VerifyCA | VerifyFull

  target:
    host: "string"          # required
    port: 5432              # required
    database: "string"      # required
    user: "string"          # required
    password: "string"      # required (use ${ENV_VAR})

  dump:
    exclude_tables:         # optional list of tables to skip entirely
      - "audit_logs"
      - "sessions"
    schema_only: false      # optional, dump structure only (no data)
    extra_pg_dump_args: []  # optional, extra flags passed to pg_dump
    parallel_jobs: 1        # optional, default: 1 — set > 1 to enable parallel dump/restore
    dump_dir: "/gcs/dumps"  # optional — persist dump and reuse it if restore fails
    timeout_minutes: 120    # optional, default: 120 — max minutes per pg_dump/pg_restore process

faker_seed: 42              # optional — fixed seed for deterministic faker output; omit for random

sanitize:
  - table: "users"          # required: target table
    where: "role != 'admin'" # optional: only update matching rows
    columns:
      - name: "email"       # required: column name
        strategy: "faker"   # required: static | null_value | template | faker | hash
        faker: "email"      # required when strategy=faker
      
      - name: "password"
        strategy: "static"
        value: "REDACTED"   # required when strategy=static

      - name: "token"
        strategy: "null_value"  # sets column to NULL

      - name: "username"
        strategy: "template"
        value: "user_{id}"  # {column_name} references resolved at runtime

      - name: "national_id"
        strategy: "hash"    # SHA256, deterministic

      - name: "phone"
        strategy: "partial_mask"
        keep_last: 4        # keep last 4 chars, mask the rest with '*'
        # keep_first: 2     # optionally also keep leading chars
        # mask_char: "*"    # optional, default: *

      - name: "status"
        strategy: "random_from"
        values: ["active", "inactive", "pending"]  # pick one at random per row

Connection Options

Field	Description	Default
host	Database hostname or IP	—
port	PostgreSQL port	—
database	Database name	—
user	Login user	—
password	Password (use env var)	—
schema	Schema to target	public
ssl_mode	SSL mode	Prefer

Safety Constraint: Source ≠ Target

The tool enforces that migration.source and migration.target cannot point to the same database (same host + port + database name). If they match, config validation will fail with an error before any data is touched. Use different database names even on the same host.

Dump Options

Field	Description	Default
exclude_tables	Tables excluded from dump	[]
schema_only	Dump DDL only, no rows	false
extra_pg_dump_args	Additional pg_dump flags	[]
parallel_jobs	Parallel workers for dump and restore (-j N). Automatically uses --format directory when > 1.	1
dump_dir	Persist dump at this path and reuse it on next run if valid. See below.	null (temp dir)
timeout_minutes	Maximum time allowed for each pg_dump / pg_restore process.	120 (2 h)

Parallel jobs

Setting parallel_jobs: 4 passes -j 4 to both pg_dump and pg_restore and automatically switches the dump format to directory (a requirement of PostgreSQL's parallel mode). The temporary directory is cleaned up after restore, just like the single-file dump.

migration:
  dump:
    parallel_jobs: 4   # uses 4 workers — good for large DBs with many tables

Parallel restore requires the dump to also have been created in directory format. If you supply a custom .dump file via other means, do not set parallel_jobs > 1.

Persistent dump directory (dump_dir)

By default db-migrator writes the dump to a temporary directory and deletes it after every run. If the restore fails you have to re-dump the entire source database from scratch.

Setting dump_dir keeps the dump alive across runs. On the next execution, if toc.dat is present and non-empty in that directory, the dump step is skipped entirely and pg_restore runs directly against the existing dump.

migration:
  dump:
    parallel_jobs: 4
    dump_dir: "/gcs/dumps/mydb"   # GCS volume mount in Cloud Run Jobs

Behavior:

Scenario	Result
dump_dir empty or toc.dat missing	Full pg_dump runs, dump saved to dump_dir
dump_dir has valid toc.dat	pg_dump skipped — ⚡ Reusing existing dump
Stale/partial dump found	Old directory deleted, re-dumped from scratch
Restore succeeds	Dump stays in dump_dir — delete manually when no longer needed

The --dump-dir <PATH> CLI flag overrides dump.dump_dir from config:

# Override config with a different path at runtime
db-migrator run --config config.yaml --dump-dir /tmp/my-dump

dump_dir always uses directory format regardless of parallel_jobs, since directory format is required for toc.dat-based validation.

Faker Seed

By default each run generates different fake values. Set faker_seed to a fixed integer to make faker output reproducible across runs:

faker_seed: 42

With a seed, the same fake values are produced as long as the config order does not change. Useful for QA environments where tests depend on specific data values.

Sanitization Strategies

Strategy	Required Extra Fields	Description
static	value	All rows get the same fixed value
null_value	—	Set column to NULL (column must be nullable)
template	value	Template with {column_ref} placeholders
faker	faker	Realistic fake data via Bogus
hash	—	Deterministic SHA256 hex string
partial_mask	keep_first and/or keep_last	Mask middle characters; preserve leading/trailing chars
random_from	values	Pick a random value from a fixed list per row

See Sanitization Strategies for detailed documentation.