Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
feat (client): support for PG on the client (#1038)
This PR adds support for Postgres on the client: - adds DB drivers for Postgres in NodeJS and in Tauri - modifies the initial Electric schema (meta tables) - modifies the queries executed by the `BundleMigrator` - modifies the triggers to also support PG - modifies the Satellite process to also support PG - modifies the CLI to bundle both SQLite and PG migrations - modifies the DAL to transform JS values to PG when writing and the other way around when reading **To test** this PR you will need to: - use a local build of Electric from `alco/vax-1659-postgresql-dialect` - npm pack the generator - use the packed generator in the ts-client - npm pack the ts-client - use the packed version of ts-client in the example app (and use node-postgres or tauri-postgres driver) Limitations: - Floating-point numbers that are out of range cannot be inserted (PG throws an error, in contrast to SQLite which silently converts them to +Inf/-Inf (for too big values) or 0 (for too small values)) - Not sure if PGlite version currently supports infinity values and/or `NaN` for floating-point numbers. Our triggers do some casting of these values, need to test this with some e2e tests or actual tauri application to see if it works or not. A couple of things to note regarding the implementation: - Had to change a trigger in both SQLite and PG for storing the JSON of a row's PK: - In PG, `jsonb` adds whitespace and re-orders keys. Whereas, SQLite has no whitespace and preserves the ordering of keys. So in PG we use `json` instead of `jsonb` and strip any whitespace but that has the side effect of also removing `null` values. Hence, we also remove `null` values in SQLite. - Postgres requires migrations coming from Electric to define all FKs as `DEFERRABLE` - We no longer defer foreign key checks before applying an incoming transaction as PG does not like deferred constraints when trying to apply some DDL. This happens because PG wants to run all trigger events before applying DDL but they are deferred so they are only allowed to run when committing so PG is stuck. So if a TX contains DML and DDL then when it encounters the DDL we get errors like this: `cannot ALTER TABLE "parent" because it has pending trigger events` Remains to be done (perhaps in a follow-up PR): - e2e tests for PG client ------------ For reviewers: - The crux of this PR consists of modifying the TS-client to abstract away dialect-specific SQL queries behind a query builder abstraction: - `src/migrators/query-builder/builder.ts` defines an abstract `QueryBuilder` class that every dialect must implement - `src/migrators/query-builder/sqliteBuilder.ts` and ``src/migrators/query-builder/pgBuilder.ts` implement the abstract class for SQLite and Postgres respectively - When electrifying a database, the dialect can (optionally) be provided in the config object. It defaults to `'SQLite'`. Postgres drivers' `electrify` function pass `'Postgres'` dialect through this config object. The DAL checks the dialect in order to apply the right conversions from JS values to SQLite/PG values and back. - The main changes are in `src/migrators/bundle.ts`, `src/migrators/schema.ts`, and `src/migrators/triggers.ts` to use the dialect-specific query builder object. - Likewise, the satellite process (`src/satellite/process.ts`) has been changed to use the dialect-specific query builder for constructing its specialised queries. - Another important change consists of supporting data types such as booleans, dates, json, etc. - In SQLite such values were converted to strings and stored as strings (since SQLite has no real support for them). - Postgres has support for those data types so we want to leverage that. - Again, we abstracted away the data type conversions behind a `Converter` interface (`src/client/conversions/converter.ts`). - Then we created 2 specific converters, 1 for SQLite `src/client/conversions/sqlite.ts` and 1 for PG `src/client/conversions/postgres.ts` - Similar to the value conversions, we also had to change serialisation/deserialisation of values in the satellite client. Currently it was encoding SQLite values on the wire and decoding values from the wire into SQLite values. We abstracted those encoders/decoders and created 2 instances of each, one for SQLite and one for PG (cf. `src/util/encoders`). --------- Co-authored-by: Oleksii Sholik <oleksii@sholik.dev> Co-authored-by: msfstef <stefanos@electric-sql.com> Co-authored-by: Sam Willis <sam.willis@gmail.com> Co-authored-by: msfstef <msfstef@gmail.com>
- Loading branch information
5 people
committed
May 2, 2024
1 parent
c8e6981
commit 450a65b