Optimize postgres validations

[rkistner]

Background

We run per-table validations for Postgres, both for the validation API (using the passed-in sync config), and the diagnostics API (using the current sync config).

This runs these checks sequentially for each table:

1. Check whether the table exists (query pg_class).
2. Check the table replica identity - default, index, full, or nothing (query pg_class).
3. Find the effective replica identity columns (query pg_attribute & pg_index).
4. Check that we can read the table - no permission errors (SELECT 1 FROM <table> LIMIT 1).
5. Check that the table is in the publication (query pg_publication_tables).
6. Check RLS config for the table (query pg_class, pg_roles).

This results in 6 sequential queries for every table. This is fine for a small number of tables, or when we have low latency to the source database. However, we have seen cases where:

1. 70+ tables are referenced in the sync config.
2. The latency to the source database is 150-200ms.

This then results in the validation and diagnostics APIs taking longer than the 60s timeout to complete.

The fix

This refactors all these checks into a single plpgsql script, using a total of 3 round-trips to the source database.

This also adds more comprehensive tests to check that we're covering each case.

For the most part, the behavior is kept the same. One exception is REPLICA INDENTITY NOTHING - this was previously reported as an error, but we do support it in the replication code (with limits - postgres prevents you from updating or deleting rows), so this check is relaxed now.

Testing

Testing validation with a high latency db connection (200ms+), 70 tables.

Before: 80s.
After: 6s, of which around 2s is from getDebugTablesInfo. The rest is from other connection checks and fetching of the schema.

The implementation

The actual queries are to a large extent based on the old ones, ported by Codex to a single plpgsql script. Since these scripts / DO blocks can't take in parameters directly, this uses set_config / current_setting to set the parameters in a local transaction.

Instead of a plpgsql script like this, it could be feasible to runs this as a couple of separate queries. The biggest difficultly would be the loop doing SELECT 1 FROM <table> LIMIT 1, which is tricky without a script (or individual queries, which would be too slow).

[changeset-bot]

🦋 Changeset detected

Latest commit: cbdc4d3

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 12 packages

Name	Type
@powersync/service-module-postgres	Patch
@powersync/service-image	Patch
@powersync/service-schema	Patch
@powersync/service-core	Patch
@powersync/service-core-tests	Patch
@powersync/service-module-core	Patch
@powersync/service-module-mongodb-storage	Patch
@powersync/service-module-mongodb	Patch
@powersync/service-module-mssql	Patch
@powersync/service-module-mysql	Patch
@powersync/service-module-postgres-storage	Patch
test-client	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[Rentacookie]

Thats a really big improvement in execution time! 🎉