fix(migrate): large-database & managed-Postgres hardening for export/import/copy#39
Merged
Merged
Conversation
Bulk inserts batched 1000 rows regardless of table width, but a multi-row insert binds rows*columns parameters. A wide table like book_metadata (66 columns) overflowed PostgreSQL's 65535 and SQLite's 32766 per-statement parameter limits during copy/import, failing with "too many arguments for query: 66000". Cap each table's batch so rows*columns stays under the destination backend's limit. Adds a wide-table reproduction (1000 book_metadata rows into PostgreSQL) and unit tests for the cap calculation.
copy/import disabled FK enforcement with `SET session_replication_role = replica`, which requires a superuser. On managed PostgreSQL the app role is only the database owner, so it failed with "permission denied to set parameter session_replication_role". On PostgreSQL, drop the FK constraints before the load and recreate them afterward — recreating an FK revalidates the loaded rows, so referential integrity is still checked, and it needs only table ownership. SQLite keeps its deferred-FK check. Runs inside the same transaction, so a failure rolls back cleanly. Adds a test that performs the copy as a non-superuser database owner.
Add a --progress flag to export/import/copy that logs per-table start and finish plus a periodic within-table row count. It's emitted as log lines (not a TTY bar) so it reads the same in an interactive terminal and in captured Kubernetes/CI logs. Verify row counts by default after import and copy: independently re-count the source (copy) or read the export manifest's recorded counts (import), count the target, compare per table, and fail the command on any mismatch. A --no-verify flag skips it.
Add an opt-in --full-verification that compares every record's canonical content on both sides and prints a report of any differences. It normalizes representation differences that are semantically irrelevant across engines: integer-valued floats (1.0 == 1), JSON object key order (jsonb reorders keys), and timestamp precision (PostgreSQL truncates to microseconds). Each table is reduced to an order-independent digest computed identically from a database connection or the archive's NDJSON, so the check streams and stays O(1) memory. The report is informational and does not fail the command; the default row-count check remains the hard safety gate. Adds canonicalization unit tests and a cross-engine test that confirms matching content and detects a tampered row.
…dling - document --progress, --no-verify, and the --full-verification report on the export/import/copy page, and note row-count verification runs by default - correct the PostgreSQL privileges note: the load now drops and recreates FK constraints (owner-only), so no superuser is required, including on managed PostgreSQL - update the contributor page for the FK approach, batch-size capping, and the two verification levels
Deploying codex with
|
| Latest commit: |
84cf1de
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://121e870d.codex-asm.pages.dev |
| Branch Preview URL: | https://fix-migrate-large-db-and-har.codex-asm.pages.dev |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Hardens the
export/import/copydatabase tooling for real-world production databases, driven by issues hit migrating a live SQLite instance to a managed PostgreSQL. Fixes two hard blockers and adds progress + verification.Changes
rows × columnsparameters; a wide table (book_metadata, ~66 columns) overflowed PostgreSQL's 65535 (and SQLite's 32766) limit and failed withtoo many arguments for query: 66000. Batches are now capped per table by the destination's limit.SET session_replication_role = replica, which requires a superuser and fails on managed PostgreSQL. It now drops and recreates the FK constraints, which needs only table ownership. Recreating an FK revalidates the loaded rows, so referential integrity is still checked — a guarantee the old path did not provide.--progresson export/import/copy: per-table start/finish plus a periodic row count, emitted as log lines so it reads the same in a terminal and in captured logs.--no-verifyskips it.--full-verification(opt-in): compares every record's canonical content on both sides and prints a report (does not fail the command). Normalizes representation differences that don't change meaning — integer-valued floats (1.0==1), JSON object key order (jsonb reorders), and timestamp precision (Postgres truncates to microseconds). Streams rows, so it stays memory-cheap.Notes
Testing
PostgreSQL-gated integration tests (skip when no test DB is reachable) cover: the wide-table batch overflow, a copy performed as a non-superuser database owner, the SQLite↔PostgreSQL round-trip, an export/import matrix across every engine pair, and full verification (matching content cross-engine plus detecting a tampered row). All green against a live PostgreSQL;
clippy -D warningsclean.