From 3f2baa9d62e3b4ca87604629d8f297673b07e49e Mon Sep 17 00:00:00 2001 From: benitav Date: Mon, 13 Jan 2025 11:50:30 +0200 Subject: [PATCH 1/9] Add Postgres bucket storage option --- .../installation/powersync-service-setup.mdx | 56 +++++++++++++++++-- 1 file changed, 50 insertions(+), 6 deletions(-) diff --git a/self-hosting/installation/powersync-service-setup.mdx b/self-hosting/installation/powersync-service-setup.mdx index 23b6dd8a..2384e50a 100644 --- a/self-hosting/installation/powersync-service-setup.mdx +++ b/self-hosting/installation/powersync-service-setup.mdx @@ -23,9 +23,13 @@ Examples of the above can be found in our demo application [here](https://github **Deploy PowerSync on Coolify:** See our [integration guide](/integration-guides/coolify) for deploying the PowerSync Service on Coolify. This can simplify the setup and management of the deployment. -## Configure MongoDB +## Configure Storage Backend -The PowerSync Service uses [MongoDB](https://www.mongodb.com/) under the hood and requires at least one replica set node. A single node is fine for development/staging environments, but a 3-node replicas set is recommended [for production](/self-hosting/lifecycle-maintenance/server-specs). +The PowerSync Service requires a storage backend for sync buckets. You can use either MongoDB or Postgres for this purpose. + +### MongoDB Storage + +MongoDB requires at least one replica set node. A single node is fine for development/staging environments, but a 3-node replicas set is recommended [for production](/self-hosting/lifecycle-maintenance/server-specs). [MongoDB Atlas](https://www.mongodb.com/products/platform/atlas-database) enables replica sets by default for new clusters. @@ -50,6 +54,33 @@ If you are rolling your own Docker environment, you can include this init script - 'sleep 10 && mongosh --host mongo:27017 --eval ''try{rs.status().ok && quit(0)} catch {} rs.initiate({_id: "rs0", version: 1, members: [{ _id: 0, host : "mongo:27017" }]})''' ``` +### Postgres Storage + +Postgres can also be used as a storage backend. This requires a separate Postgres instance from your data source database. + +#### Database Setup + +You'll need to create a user and schema for PowerSync storage. You can either: + +1. Let PowerSync create the schema (recommended): + +```sql +CREATE USER powersync_storage_user WITH PASSWORD 'secure_password'; +-- The user should only have access to the schema it created +GRANT CREATE ON DATABASE postgres TO powersync_storage_user; +``` + +2. Or manually create the schema: + +```sql +CREATE USER powersync_storage_user WITH PASSWORD 'secure_password'; +CREATE SCHEMA IF NOT EXISTS powersync AUTHORIZATION powersync_storage_user; +GRANT CONNECT ON DATABASE postgres TO powersync_storage_user; +GRANT USAGE ON SCHEMA powersync TO powersync_storage_user; +GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA powersync TO powersync_storage_user; +``` + + ## PowerSync Configuration The PowerSync Service is configured using key/value pairs in a config file, and supports the following configuration methods: @@ -70,7 +101,7 @@ The config file schema is also available here: -Below is a skeleton config file that you can copy/paste and edit locally: +Below is a skeleton config file you can copy and paste to edit locally: ```yaml # config.yaml @@ -92,14 +123,23 @@ replication: sslmode: disable # 'verify-full' (default) or 'verify-ca' or 'disable' # 'disable' is OK for local/private networks, not for public networks -# Connection settings for sync bucket storage +# Connection settings for sync bucket storage (Postgres and MongoDB are supported) storage: + # Option 1: MongoDB Storage type: mongodb uri: mongodb://mongo:27017/powersync_demo - # use these if authentication is required # username: myuser # password: mypassword + # Option 2: Postgres Storage + # type: postgresql + # uri: postgresql://powersync_storage_user:secure_password@storage-db:5432/postgres + # Optional batch limits configuration + # batch_limits: + # max_estimated_size: 5000000 + # max_record_count: 2000 + # max_current_data_batch_size: 50000000 + # The port which the PowerSync API server will listen on port: 80 @@ -132,6 +172,10 @@ client_auth: ``` + +**Important:** When using Postgres storage, you must use a separate database instance from your data source database. Using the same database for both purposes may cause unexpected results. + + Specify the connection to Postgres in the `replication` section. Retrieving your database connection string / individual parameters differs by database hosting provider. See [Database Connection](/self-hosting/appendix/database-connection) for further details. @@ -142,7 +186,7 @@ Specify the connection to Postgres in the `replication` section. Retrieving your This is because Supabase only allows direct database connections over IPv6 — PowerSync cannot connect using the connection pooler. -Specify the connection to MongoDB in the `storage` section. +Specify the connection to your sync bucket storage provider (Postgres or MongoDB) in the `storage` section. ### Environment Variables From 97eb117de0e46403e4e602bf4e921f76e19f6fa2 Mon Sep 17 00:00:00 2001 From: benitav Date: Mon, 13 Jan 2025 12:19:08 +0200 Subject: [PATCH 2/9] Simplify skeleton config --- self-hosting/installation/powersync-service-setup.mdx | 11 ++++------- 1 file changed, 4 insertions(+), 7 deletions(-) diff --git a/self-hosting/installation/powersync-service-setup.mdx b/self-hosting/installation/powersync-service-setup.mdx index 2384e50a..8311316c 100644 --- a/self-hosting/installation/powersync-service-setup.mdx +++ b/self-hosting/installation/powersync-service-setup.mdx @@ -80,7 +80,6 @@ GRANT USAGE ON SCHEMA powersync TO powersync_storage_user; GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA powersync TO powersync_storage_user; ``` - ## PowerSync Configuration The PowerSync Service is configured using key/value pairs in a config file, and supports the following configuration methods: @@ -123,22 +122,20 @@ replication: sslmode: disable # 'verify-full' (default) or 'verify-ca' or 'disable' # 'disable' is OK for local/private networks, not for public networks -# Connection settings for sync bucket storage (Postgres and MongoDB are supported) +# Connection settings for sync bucket storage (MongoDB and Postgres are supported) storage: # Option 1: MongoDB Storage type: mongodb uri: mongodb://mongo:27017/powersync_demo + # Use these if authentication is required. The user should have `readWrite` and `dbAdmin` roles # username: myuser # password: mypassword # Option 2: Postgres Storage # type: postgresql + # This accepts the same parameters as a Postgres replication source connection # uri: postgresql://powersync_storage_user:secure_password@storage-db:5432/postgres - # Optional batch limits configuration - # batch_limits: - # max_estimated_size: 5000000 - # max_record_count: 2000 - # max_current_data_batch_size: 50000000 + # sslmode: disable # 'verify-full' (default) or 'verify-ca' or 'disable' # The port which the PowerSync API server will listen on port: 80 From 321a7f1cc6440299c940c9d9dea5bd8cc075f24d Mon Sep 17 00:00:00 2001 From: benitav Date: Mon, 13 Jan 2025 12:22:40 +0200 Subject: [PATCH 3/9] Improve wording --- self-hosting/installation/powersync-service-setup.mdx | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/self-hosting/installation/powersync-service-setup.mdx b/self-hosting/installation/powersync-service-setup.mdx index 8311316c..c7a049ed 100644 --- a/self-hosting/installation/powersync-service-setup.mdx +++ b/self-hosting/installation/powersync-service-setup.mdx @@ -56,7 +56,7 @@ If you are rolling your own Docker environment, you can include this init script ### Postgres Storage -Postgres can also be used as a storage backend. This requires a separate Postgres instance from your data source database. +Postgres can also be used as a storage backend. This requires a separate Postgres instance from your replication source database. #### Database Setup @@ -170,7 +170,7 @@ client_auth: ``` -**Important:** When using Postgres storage, you must use a separate database instance from your data source database. Using the same database for both purposes may cause unexpected results. +**Important:** When using Postgres storage, you must use a separate database instance from your replication source database. Using the same database for both purposes may cause unexpected results. Specify the connection to Postgres in the `replication` section. Retrieving your database connection string / individual parameters differs by database hosting provider. See [Database Connection](/self-hosting/appendix/database-connection) for further details. From ff768a1d04efb797ad3d06a63a1ec78a14c506a8 Mon Sep 17 00:00:00 2001 From: benitav Date: Mon, 13 Jan 2025 12:31:22 +0200 Subject: [PATCH 4/9] Improve wording --- .../installation/powersync-service-setup.mdx | 24 +++++++++++-------- 1 file changed, 14 insertions(+), 10 deletions(-) diff --git a/self-hosting/installation/powersync-service-setup.mdx b/self-hosting/installation/powersync-service-setup.mdx index c7a049ed..fec0e84f 100644 --- a/self-hosting/installation/powersync-service-setup.mdx +++ b/self-hosting/installation/powersync-service-setup.mdx @@ -3,15 +3,15 @@ title: "PowerSync Service Setup" description: "Configuration details for connecting the PowerSync Service to your database" --- -After configuring your Postgres database for PowerSync, you'll setup your [PowerSync Service](/architecture/powersync-service). +After configuring your source database for PowerSync, you'll need to setup your [PowerSync Service](/architecture/powersync-service). This entails: -1. Configuring MongoDB (if required) +1. Configuring sync bucket storage 2. Defining your PowerSync config - 1. Defining connections to Postgres and MongoDB + 1. Defining connections to your source database, and sync bucket storage database 2. Defining your [Sync Rules](/usage/sync-rules) @@ -25,11 +25,11 @@ Examples of the above can be found in our demo application [here](https://github ## Configure Storage Backend -The PowerSync Service requires a storage backend for sync buckets. You can use either MongoDB or Postgres for this purpose. +The PowerSync Service requires a storage backend for sync buckets. You can use either MongoDB or Postgres for this purpose. The storage backend is separate from your source database. ### MongoDB Storage -MongoDB requires at least one replica set node. A single node is fine for development/staging environments, but a 3-node replicas set is recommended [for production](/self-hosting/lifecycle-maintenance/server-specs). +MongoDB requires at least one replica set node. A single node is fine for development/staging environments, but a 3-node replica set is recommended [for production](/self-hosting/lifecycle-maintenance/server-specs). [MongoDB Atlas](https://www.mongodb.com/products/platform/atlas-database) enables replica sets by default for new clusters. @@ -56,11 +56,11 @@ If you are rolling your own Docker environment, you can include this init script ### Postgres Storage -Postgres can also be used as a storage backend. This requires a separate Postgres instance from your replication source database. +Postgres can also be used as a storage backend. #### Database Setup -You'll need to create a user and schema for PowerSync storage. You can either: +You'll need to create a dedicated user and schema for PowerSync storage. You can either: 1. Let PowerSync create the schema (recommended): @@ -80,6 +80,10 @@ GRANT USAGE ON SCHEMA powersync TO powersync_storage_user; GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA powersync TO powersync_storage_user; ``` + +**Demo app:** A demo app with Postgres bucket storage is available [here](TODO). + + ## PowerSync Configuration The PowerSync Service is configured using key/value pairs in a config file, and supports the following configuration methods: @@ -120,7 +124,7 @@ replication: # SSL settings sslmode: disable # 'verify-full' (default) or 'verify-ca' or 'disable' - # 'disable' is OK for local/private networks, not for public networks + # Note: 'disable' is only suitable for local/private networks, not for public networks # Connection settings for sync bucket storage (MongoDB and Postgres are supported) storage: @@ -135,7 +139,7 @@ storage: # type: postgresql # This accepts the same parameters as a Postgres replication source connection # uri: postgresql://powersync_storage_user:secure_password@storage-db:5432/postgres - # sslmode: disable # 'verify-full' (default) or 'verify-ca' or 'disable' + # sslmode: disable # The port which the PowerSync API server will listen on port: 80 @@ -170,7 +174,7 @@ client_auth: ``` -**Important:** When using Postgres storage, you must use a separate database instance from your replication source database. Using the same database for both purposes may cause unexpected results. +**Important:** When using Postgres storage, you must use a separate database instance from your source database used for replication. Using the same database for both purposes may cause unexpected behavior. Specify the connection to Postgres in the `replication` section. Retrieving your database connection string / individual parameters differs by database hosting provider. See [Database Connection](/self-hosting/appendix/database-connection) for further details. From e5b86e063ec8152ce943a1d27e603a11f8dccdcb Mon Sep 17 00:00:00 2001 From: benitav Date: Mon, 13 Jan 2025 13:02:01 +0200 Subject: [PATCH 5/9] Update references to MongoDB storage --- self-hosting/installation.mdx | 2 +- self-hosting/installation/powersync-service-setup.mdx | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/self-hosting/installation.mdx b/self-hosting/installation.mdx index cc2c1ce1..0339932d 100644 --- a/self-hosting/installation.mdx +++ b/self-hosting/installation.mdx @@ -29,7 +29,7 @@ In order to run the PowerSync Service, the following activities are required: - + diff --git a/self-hosting/installation/powersync-service-setup.mdx b/self-hosting/installation/powersync-service-setup.mdx index fec0e84f..d6d2f6e2 100644 --- a/self-hosting/installation/powersync-service-setup.mdx +++ b/self-hosting/installation/powersync-service-setup.mdx @@ -23,7 +23,7 @@ Examples of the above can be found in our demo application [here](https://github **Deploy PowerSync on Coolify:** See our [integration guide](/integration-guides/coolify) for deploying the PowerSync Service on Coolify. This can simplify the setup and management of the deployment. -## Configure Storage Backend +## Configure Sync Bucket Storage The PowerSync Service requires a storage backend for sync buckets. You can use either MongoDB or Postgres for this purpose. The storage backend is separate from your source database. From 5d39e85ae7bd528b37562455b5295d0502cc8d33 Mon Sep 17 00:00:00 2001 From: benitav Date: Mon, 13 Jan 2025 13:12:15 +0200 Subject: [PATCH 6/9] Fix some missing MySQL references --- installation/app-backend-setup.mdx | 2 +- installation/app-backend-setup/writing-client-changes.mdx | 2 +- .../client-side-setup/integrating-with-your-backend.mdx | 2 +- resources/demo-apps-example-projects.mdx | 5 +++-- usage/sync-rules.mdx | 2 +- usage/sync-rules/advanced-topics/sharded-databases.mdx | 2 +- 6 files changed, 8 insertions(+), 7 deletions(-) diff --git a/installation/app-backend-setup.mdx b/installation/app-backend-setup.mdx index 4b0eb9be..6cd8b2db 100644 --- a/installation/app-backend-setup.mdx +++ b/installation/app-backend-setup.mdx @@ -7,7 +7,7 @@ When using PowerSync, your app backend is responsible for the following: 1. Authenticating app users 2. [Generating JWTs](/installation/authentication-setup) that allows the PowerSync Client SDK to authenticate users against the server-side [PowerSync Service](/architecture/powersync-service). -3. [Writing client-side changes](/installation/app-backend-setup/writing-client-changes) to the backend database (Postgres or MongoDB) +3. [Writing client-side changes](/installation/app-backend-setup/writing-client-changes) to the backend database (Postgres, MongoDB or MySQL) diff --git a/installation/app-backend-setup/writing-client-changes.mdx b/installation/app-backend-setup/writing-client-changes.mdx index 9e2e44bb..85c3d3ad 100644 --- a/installation/app-backend-setup/writing-client-changes.mdx +++ b/installation/app-backend-setup/writing-client-changes.mdx @@ -20,7 +20,7 @@ In other words, don't place writes into something like a queue for processing la -Client applications advance to a write checkpoint after uploads have been processed, so if the client believes that the server has written changes into your backend database (Postgres or MongoDB), but the next checkpoint does not contain your uploaded changes, those changes will be removed from the client. This could manifest as UI glitches for your end-users, where the changes disappear from the device for a few seconds and then re-appear. +Client applications advance to a write checkpoint after uploads have been processed, so if the client believes that the server has written changes into your backend database (Postgres, MongoDB or MySQL), but the next checkpoint does not contain your uploaded changes, those changes will be removed from the client. This could manifest as UI glitches for your end-users, where the changes disappear from the device for a few seconds and then re-appear. ### Changes recorded on the client diff --git a/installation/client-side-setup/integrating-with-your-backend.mdx b/installation/client-side-setup/integrating-with-your-backend.mdx index ed707bbc..d5ea2712 100644 --- a/installation/client-side-setup/integrating-with-your-backend.mdx +++ b/installation/client-side-setup/integrating-with-your-backend.mdx @@ -7,7 +7,7 @@ description: "The PowerSync 'backend connector' provides the connection between It is used to: 1. Retrieve a JWT token which can be used by the PowerSync Client SDK to authenticate against your [PowerSync Service](/architecture/powersync-service) instance. -2. Upload writes to your backend: Writes that are made to the local SQLite database are sent to your backend application, where you control how they're applied to your backend database (Postgres or MongoDB) +2. Upload writes to your backend: Writes that are made to the local SQLite database are sent to your backend application, where you control how they're applied to your backend database (Postgres, MongoDB or MySQL) Accordingly, the connector must implement two methods: diff --git a/resources/demo-apps-example-projects.mdx b/resources/demo-apps-example-projects.mdx index 72f0f46d..adcfd7b3 100644 --- a/resources/demo-apps-example-projects.mdx +++ b/resources/demo-apps-example-projects.mdx @@ -94,10 +94,11 @@ Example projects are listed under backend they use, but you can easily wire up y * [To-Do List App with Docker Compose](https://github.com/powersync-ja/self-host-demo) - * [Postgres + Node.js Backend](https://github.com/powersync-ja/self-host-demo/tree/main/demos/nodejs) + * [Postgres + Node.js Backend](https://github.com/powersync-ja/self-host-demo/tree/main/demos/nodejs-postgres-bucket-storage) + * [Postgres + Postgres sync bucket storage + Node.js Backend](https://github.com/powersync-ja/self-host-demo/tree/main/demos/nodejs) * [MongoDB + Node.js Backend](https://github.com/powersync-ja/self-host-demo/tree/main/demos/nodejs-mongodb) * [MySQL + Node.js Backend](https://github.com/powersync-ja/self-host-demo/tree/main/demos/nodejs-mysql) - * [Supabase Backend + Local Development Example](https://github.com/powersync-ja/self-host-demo/tree/main/demos/supabase) + * [Supabase Backend (Postgres) + Local Development Example](https://github.com/powersync-ja/self-host-demo/tree/main/demos/supabase) * [Django Backend](https://github.com/powersync-ja/self-host-demo/tree/main/demos/django) diff --git a/usage/sync-rules.mdx b/usage/sync-rules.mdx index 3af97cb3..0704346e 100644 --- a/usage/sync-rules.mdx +++ b/usage/sync-rules.mdx @@ -15,7 +15,7 @@ The remainder of these docs dive further into the details. Each [PowerSync Service](/architecture/powersync-service) instance has a Sync Rules configuration where Sync Rules are defined using SQL-like queries (limitations and more info [here](/usage/sync-rules/operators-and-functions)) combined together in a YAML file. -This SQL-like syntax is used when connecting to either Postgres or MongoDB as the backend source database. +This SQL-like syntax is used when connecting to either Postgres, MongoDB or MySQL as the backend source database. The [PowerSync Service](/architecture/powersync-service) uses these SQL-like queries to group data into "sync buckets" when replicating data to client devices. diff --git a/usage/sync-rules/advanced-topics/sharded-databases.mdx b/usage/sync-rules/advanced-topics/sharded-databases.mdx index 5a00c1e9..b9bf1201 100644 --- a/usage/sync-rules/advanced-topics/sharded-databases.mdx +++ b/usage/sync-rules/advanced-topics/sharded-databases.mdx @@ -8,7 +8,7 @@ In the case of Postgres, PowerSync cannot replicate Postgres [foreign tables](ht However, PowerSync does have options available to support sharded databases in general. - When using MongoDB as the backend source database, PowerSync does not currently support connecting to sharded clusters. + When using MongoDB or MySQL as the backend source database, PowerSync does not currently support connecting to sharded clusters. The primary options are: From 3c63c1c8d2850ebde4455640e28191cf3ff1069a Mon Sep 17 00:00:00 2001 From: benitav Date: Mon, 13 Jan 2025 13:15:49 +0200 Subject: [PATCH 7/9] Update self-hosting/installation/powersync-service-setup.mdx --- self-hosting/installation/powersync-service-setup.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/self-hosting/installation/powersync-service-setup.mdx b/self-hosting/installation/powersync-service-setup.mdx index d6d2f6e2..29a55260 100644 --- a/self-hosting/installation/powersync-service-setup.mdx +++ b/self-hosting/installation/powersync-service-setup.mdx @@ -81,7 +81,7 @@ GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA powersync TO powersync_storage_user ``` -**Demo app:** A demo app with Postgres bucket storage is available [here](TODO). +**Demo app:** A demo app with Postgres bucket storage is available [here](https://github.com/powersync-ja/self-host-demo/tree/main/demos/nodejs-postgres-bucket-storage). ## PowerSync Configuration From db1b308a748d8b490e7c6e49234c5a4976f4d216 Mon Sep 17 00:00:00 2001 From: benitav Date: Mon, 13 Jan 2025 13:22:48 +0200 Subject: [PATCH 8/9] Wording improvements --- self-hosting/lifecycle-maintenance.mdx | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/self-hosting/lifecycle-maintenance.mdx b/self-hosting/lifecycle-maintenance.mdx index e038dbb0..27e8e41b 100644 --- a/self-hosting/lifecycle-maintenance.mdx +++ b/self-hosting/lifecycle-maintenance.mdx @@ -12,14 +12,14 @@ Migrations run automatically by default. # powersync.yaml migrations: - # Setting this to true will skip automatic migrations. - # Migrations can be triggered externally by altering the container `command`. + # Setting this to false (default) enables automatic migrations on startup. + # When set to true, migrations must be triggered manually by modifying the container `command`. disable_auto_migration: true ``` -MongoDB locks are used to ensure that migrations are only executed once in the case where multiple containers can be started simultaneously. +MongoDB locks ensure migrations are executed exactly once, even when multiple containers start simultaneously. ## Backups -We recommend using git to backup your configuration files. +We recommend using Git to backup your configuration files. -Your MongoDB database (for sync bucket storage) does not need to be backed up since it's easily recreated from Postgres. +The sync bucket storage database doesn't require backups as it can be easily reconstructed. From a5ea6ad6d4f6e373fc23686e72d6d161446b517b Mon Sep 17 00:00:00 2001 From: benitav Date: Tue, 14 Jan 2025 11:59:10 +0200 Subject: [PATCH 9/9] Beta label and postgres server --- self-hosting/installation/powersync-service-setup.mdx | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/self-hosting/installation/powersync-service-setup.mdx b/self-hosting/installation/powersync-service-setup.mdx index 29a55260..a611cc65 100644 --- a/self-hosting/installation/powersync-service-setup.mdx +++ b/self-hosting/installation/powersync-service-setup.mdx @@ -54,9 +54,9 @@ If you are rolling your own Docker environment, you can include this init script - 'sleep 10 && mongosh --host mongo:27017 --eval ''try{rs.status().ok && quit(0)} catch {} rs.initiate({_id: "rs0", version: 1, members: [{ _id: 0, host : "mongo:27017" }]})''' ``` -### Postgres Storage +### Postgres Storage (Beta) -Postgres can also be used as a storage backend. +Available since version 1.3.8 of [`journeyapps/powersync-service`](https://hub.docker.com/r/journeyapps/powersync-service), you can use Postgres as an alternative storage backend for sync buckets. This feature is currently in [beta](/resources/feature-status). #### Database Setup @@ -174,7 +174,7 @@ client_auth: ``` -**Important:** When using Postgres storage, you must use a separate database instance from your source database used for replication. Using the same database for both purposes may cause unexpected behavior. +**Important:** When using Postgres for sync bucket storage, a separate server is currently required for replication connections (if using Postgres for replication) and storage. Using the same server might cause unexpected results. Specify the connection to Postgres in the `replication` section. Retrieving your database connection string / individual parameters differs by database hosting provider. See [Database Connection](/self-hosting/appendix/database-connection) for further details.