Skip to content

Add unit test coverage for scripts/migrate-selfhost-sqlite-to-postgres.ts (zero tests for a high-stakes data migration) #8391

Description

@JSONbored

Context

scripts/migrate-selfhost-sqlite-to-postgres.ts (~480 lines) is the one-shot data migration a
self-host operator runs to move their SQLite backend to Postgres (npm run selfhost:postgres:migrate).
It is high-stakes by design: it copies every table's rows across engines, validates row counts and
key-matching after copy, resets Postgres sequences, prunes schema-init seed rows, and defaults to a
transactionally-rolled-back dry run unless --execute is passed. Despite that, it has zero test
coverage
anywhere under test/** — confirmed via find test -iname "*migrate-selfhost*" /
*sqlite-to-postgres*", both empty.

This is a real gap, not a hypothetical: scripts/** is excluded from Codecov's coverage.include
(codecov.yml), so nothing in CI currently forces this file to be tested, and a regression here would
only ever surface when an operator runs a real, potentially destructive migration against their own
data.

Several of this script's exported pure functions are directly testable without a real Postgres server,
following the exact mocking convention test/unit/selfhost-pg-adapter.test.ts already established for
src/selfhost/pg-adapter.ts (a hand-built Pool/PoolClient mock whose query() records every SQL
statement + params to a shared log, so a test can assert on exact generated SQL):

  • quoteIdent (identifier-injection guard — throws on any non-[A-Za-z0-9_] identifier)
  • insertSql (batched multi-row INSERT ... VALUES (...), (...) [ON CONFLICT (...) DO NOTHING] builder)
  • valuePlaceholder (the _selfhost_vectors ::vector/::jsonb cast special-case)
  • normalizePostgresValue (NUL-byte replacement for Postgres text/json columns — SQLite text can contain
    \0, Postgres cannot)
  • parseArgs (flag parsing + the --postgres-url/DATABASE_URL scheme validation, the
    --batch-size positive-integer guard, the SQLite-source/migrations-dir existence checks)

The higher-level orchestration (copyAll, copyTable, countTargetRowsMatchingSourceRows,
countConflictingTargetRowsForSourceKeys, resetPgSequences) needs the same mock-PoolClient
approach test/unit/selfhost-pg-adapter.test.ts/test/unit/selfhost-pg-queue.test.ts already use, paired
with a real in-memory node:sqlite DatabaseSync source (the script already imports
node:sqlite's DatabaseSync, and the repo already relies on in-memory node:sqlite elsewhere —
test/helpers/d1.ts's TestD1Database).

Requirements

  • Add test/unit/migrate-selfhost-sqlite-to-postgres.test.ts (or a -script suffix matching this
    repo's existing naming for CLI-wrapper test files, e.g. selfhost-env-reference-script.test.ts).
  • Cover the pure functions listed above directly (import them from the script — they are already
    top-level exported or straightforward to export if currently unexported; check current export
    status and add export only where genuinely needed for testability, not a broad refactor).
  • Cover copyAll's real decision branches using a mock PoolClient (mirroring
    test/unit/selfhost-pg-adapter.test.ts's makeMockPool pattern) + a real in-memory DatabaseSync
    loaded with a couple of tiny fixture tables:
    • happy path: empty target, common columns present, rows copied, sequences reset.
    • _selfhost_vectors skipped without --include-vectors, copied with it.
    • throws when the target Postgres schema is missing a source table.
    • throws when the target already has rows and --allow-non-empty is not set.
    • --allow-non-empty with a non-conflicting key overlap succeeds; with a conflicting one throws.
    • throws when a non-empty target table has no comparable primary key overlap.
    • the post-copy validation branch (targetCount < targetRowsBefore throws; the
      _selfhost_job_stats "at least" comparison vs. the exact-match comparison for other tables).
  • Cover parseArgs's validation branches (missing/invalid --postgres-url, non-numeric/<1
    --batch-size, missing SQLite source file, missing migrations dir, --help exit).

Deliverables

  • test/unit/migrate-selfhost-sqlite-to-postgres.test.ts (or equivalently named) covering the
    functions and branches listed above.
  • Any minimal export additions to scripts/migrate-selfhost-sqlite-to-postgres.ts needed to make
    the above testable, with no behavior change to the script itself.

Test Coverage Requirements

scripts/** is excluded from Codecov's coverage.include (codecov.yml), so this PR owes no
codecov/patch percentage — there is no automated coverage gate for this path today, which is exactly
the gap this issue closes. Verification instead is: the new test file passes under
npx vitest run test/unit/<new-file>.test.ts, exercises every branch listed in Requirements (both the
throw and success side of each), and the full npm run test:coverage run stays green. Do not add a
scripts/** entry to codecov.yml's include as part of this change — that's a separate,
repo-wide decision out of scope here.

Expected Outcome

A regression in scripts/migrate-selfhost-sqlite-to-postgres.ts's SQL-generation, validation, or
conflict-detection logic is caught by npm run test:coverage locally and in CI, instead of only being
discoverable by an operator running a real (potentially destructive) Postgres migration.

Links & Resources

  • scripts/migrate-selfhost-sqlite-to-postgres.ts (the file needing tests)
  • test/unit/selfhost-pg-adapter.test.ts (the mock-Pool/PoolClient pattern to mirror)
  • test/unit/selfhost-pg-queue.test.ts / test/unit/selfhost-pg-vectorize.test.ts (siblings using the
    same mock convention)
  • test/helpers/d1.ts (existing in-memory node:sqlite usage pattern in this repo's test suite)
  • codecov.yml (confirms scripts/** is excluded from coverage.include)

Metadata

Metadata

Assignees

No one assigned

    Labels

    gittensor:bugGittensor-scored bug fix — scores a 0.05x multiplier.help wantedExtra attention is needed

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions