Skip to content

Commit

Permalink
First draft PG client docs
Browse files Browse the repository at this point in the history
  • Loading branch information
samwillis committed May 1, 2024
1 parent b7dcace commit d5dda94
Show file tree
Hide file tree
Showing 18 changed files with 195 additions and 27 deletions.
6 changes: 3 additions & 3 deletions docs/api/clients/typescript.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,19 +10,19 @@ sidebar_position: 10
The Typescript client provides a number of functions for developing front-end applications with Electric:

- [Authenticating](../../usage/auth/) with the sync service
- [Synchronising database](#shapes) to a local SQLite database
- [Synchronising database](#shapes) to a local SQLite, PGlite or Postgres database
- [Type-safe data access](#queries) to read and update the database
- [Reactive live queries](#live-queries) that update in realtime as the database changes

## Instantiation

A Typescript client comprises of:

1. SQLite database connection from a [supported driver](../../integrations/drivers/)
1. A SQLite, PGlite or Postgres database connection from a [supported driver](../../integrations/drivers/)
2. A client schema [generated using the generator command](../cli.md#generate)
3. A [configuration object](#configuration)

To instantiate the client, these are passed to an `electrify` function that is specific to your SQLite database driver and platform.
To instantiate the client, these are passed to an `electrify` function that is specific to your database driver and platform.

```ts
import { schema } from './generated/client'
Expand Down
2 changes: 1 addition & 1 deletion docs/integrations/drivers/_category_.yml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
customProps:
description: >-
Adapt the SQLite database driver for your target environment.
Adapt the SQLite or Postgres database driver for your target environment.
hide_overview: true
icon: /img/icons/nodejs.png
link:
Expand Down
4 changes: 2 additions & 2 deletions docs/integrations/drivers/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,11 +5,11 @@ sidebar_position: 10

import DocCardList from '@theme/DocCardList';

Adapt the SQLite database driver for your target environment.
Adapt the local SQLite, PGlite or Postgres database driver for your target environment.

## How drivers work

ElectricSQL works with existing SQLite database drivers. Connect to your local SQLite database using your existing driver library. Then pass the database `conn` to your driver adapter's `electrify` function when [instantiating your Client](../../usage/data-access/client.md#instantiating-the-client).
The ElectricSQL client works with existing SQLite and Postgres database drivers as well as [PGlite](http://github.com/electric-sql/pglite/), our WASM build of Postgres. Connect to your local database using your existing driver library. Then pass the database `conn` to your driver adapter's `electrify` function when [instantiating your Client](../../usage/data-access/client.md#instantiating-the-client).

## Supported drivers

Expand Down
2 changes: 1 addition & 1 deletion docs/integrations/drivers/other/_category_.yml
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
label: Other
customProps:
description: >-
Create your own SQLite driver adapter.
Create your own SQLite or Postgres driver adapter.
hide_overview: true
icon: /img/icons/other.svg
position: 40
Expand Down
2 changes: 1 addition & 1 deletion docs/integrations/drivers/server/_category_.yml
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
label: Server
customProps:
description: >-
SQLite driver adapters for the server.
SQLite and Postgres driver adapters for NodeJS on the server.
hide_overview: true
icon: /img/icons/nodejs.png
position: 30
Expand Down
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
---
title: NodeJS
title: better-sqlite3
---

ElectricSQL supports [Node.js](https://nodejs.org) server application using the [better-sqlite3](https://github.com/WiseLibs/better-sqlite3) driver.
Expand Down
53 changes: 53 additions & 0 deletions docs/integrations/drivers/server/pglite.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
---
title: PGlite
---

ElectricSQL supports [Node.js](https://nodejs.org) server application using using [PGlite](https://github.com/electric-sql/pglite/), our lightweight WebAssembly build of Postgres.

## Dependencies

Add `@electric-sql/pglite` as a dependency to your app, e.g.:

```shell
npm install @electric-sql/pglite
```

See the [PGlite repo](https://github.com/electric-sql/pglite/) for more information.

## Usage

```tsx
import { electrify } from 'electric-sql/pglite'
import { PGlite } from '@electric-sql/pglite'

// Import your generated database schema.
import { schema } from './generated/client'

// Define custom configuration if needed
const config = {
url: 'https://example.com:5133'
}

// Create the PGlite database connection. The first argument
// is your database name. Changing this will create/use a new
// local database file.
const conn = new PGlite('/path/to/postgres/datadir')

// Instantiate your electric client.
const electric = await electrify(conn, schema, config)

// Connect to Electric, passing along your authentication token
// See Usage -> Authentication for more details.
await electric.connect('your token')
```

You can now use the client to read, write and sync data, e.g.:

```tsx
const { db } = electric

const results = await db.projects.findMany()
console.log(results)
```

See <DocPageLink path="usage/data-access" /> and <DocPageLink path="integrations/frontend" /> for more information.
54 changes: 54 additions & 0 deletions docs/integrations/drivers/server/postgres.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
---
title: node-postgres
---

ElectricSQL supports [Node.js](https://nodejs.org) server application using the [node-postgres](https://node-postgres.com) driver.

## Dependencies

Add `pg` as a dependency to your app, e.g.:

```shell
npm install pg
```

## Usage

```tsx
import pg from 'pg'
import { electrify, ElectricDatabase } from 'electric-sql/node-postgres'

// Import your generated database schema.
import { schema } from './generated/client'

// Define custom configuration if needed
const config = {
url: 'https://example.com:5133'
}

// Create the node-postgres database connection.
const client = new pg.Client({
// Connection configuration, see:
// https://node-postgres.com/apis/client
})
await client.connect()

// Instantiate your electric client.
const conn = ElectricDatabase(client, 'name-of-database')
const electric = await electrify(conn, schema, config)

// Connect to Electric, passing along your authentication token
// See Usage -> Authentication for more details.
await electric.connect('your token')
```

You can now use the client to read, write and sync data, e.g.:

```tsx
const { db } = electric

const results = await db.projects.findMany()
console.log(results)
```

See <DocPageLink path="usage/data-access" /> and <DocPageLink path="integrations/frontend" /> for more information.
2 changes: 1 addition & 1 deletion docs/integrations/drivers/web/_category_.yml
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
label: Web
customProps:
description: >-
SQLite driver adapters for the web.
SQLite and Postgres driver adapters for the web.
hide_overview: true
icon: /img/icons/firefox.svg
position: 10
Expand Down
61 changes: 61 additions & 0 deletions docs/integrations/drivers/web/pglite.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
---
title: PGlite
---

ElectricSQL supports running in the web browser using [PGlite](https://github.com/electric-sql/pglite/), our lightweight WebAssembly build of Postgres that supports in-browser persistence using IndexedDB.

## Dependencies

Add `@electric-sql/pglite` as a dependency to your app, e.g.:

```shell
npm install @electric-sql/pglite
```

See the [PGlite repo](https://github.com/electric-sql/pglite/) for more information.

## Usage

```tsx
import { electrify } from 'electric-sql/pglite'
import { PGlite } from '@electric-sql/pglite'

// Import your generated database schema.
import { schema } from './generated/client'

// Define custom configuration if needed
const config = {
url: 'https://example.com:5133'
}

// Create the PGlite database connection. The first argument
// is your database name. Changing this will create/use a new
// local database file.
// PGlite uses a `idb://` prefix to specify that the database
// is stored in indexedDB.
const conn = new PGlite('idb://electric.db', {
// You can optionally use the relaxed durability mode to
// improve responsiveness.
// This schedules flush to indexedDB for after a query has
// returned.
relaxedDurability: true,
})

// Instantiate your electric client.
const electric = await electrify(conn, schema, config)

// Connect to Electric, passing along your authentication token
// See Usage -> Authentication for more details.
await electric.connect('your token')
```

You can now use the client to read, write and sync data, e.g.:

```tsx
const { db } = electric

const results = await db.projects.findMany()
console.log(results)
```

See the [examples/web-wa-sqlite](https://github.com/electric-sql/electric/tree/main/examples/web-wa-sqlite) demo app, <DocPageLink path="usage/data-access" /> and <DocPageLink path="integrations/frontend" /> for more information.
2 changes: 1 addition & 1 deletion docs/quickstart/_setup_manual.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ See <DocPageLink path="usage/installation/service" /> for more info.
### Typescript client
Add the `electric-sql` library to your web or mobile app, along with an [SQLite driver](../integrations/drivers/index.md):
Add the `electric-sql` library to your web or mobile app, along with an [SQLite or Postgres driver](../integrations/drivers/index.md):
```shell
npm install electric-sql
Expand Down
2 changes: 1 addition & 1 deletion docs/quickstart/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -76,7 +76,7 @@ ELECTRIC GRANT ALL

### Instantiate

Wrap your [SQLite driver](../integrations/drivers/index.md) with a type-safe, schema-aware [database Client](../usage/data-access/client.md):
Wrap your local [SQLite or Postgres driver](../integrations/drivers/index.md) with a type-safe, schema-aware [database Client](../usage/data-access/client.md):

```tsx
import { electrify, ElectricDatabase } from 'electric-sql/wa-sqlite'
Expand Down
12 changes: 6 additions & 6 deletions docs/reference/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ ElectricSQL has three primary components. [Postgres](../usage/installation/postg

[![High level component diagramme](./_images/high-level-components.png)](./_images/high-level-components.jpg)

Inside the Client there is a [generated](../api/cli.md#generate) type-safe [data access library](../usage/data-access/client.md). A [satellite replication process](https://github.com/electric-sql/electric/blob/main/clients/typescript/src/satellite/process.ts) (per named local database), SQLite [driver adapters](../integrations/drivers/index.md) and a [reactivity system](#reactivity) with [front-end framework integrations](../integrations/frontend/index.md).
Inside the Client there is a [generated](../api/cli.md#generate) type-safe [data access library](../usage/data-access/client.md). A [satellite replication process](https://github.com/electric-sql/electric/blob/main/clients/typescript/src/satellite/process.ts) (per named local database), SQLite or Postgres [driver adapters](../integrations/drivers/index.md) and a [reactivity system](#reactivity) with [front-end framework integrations](../integrations/frontend/index.md).

### Topology

Expand Down Expand Up @@ -50,7 +50,7 @@ This library and the migrations are imported into the local app. When the app [i

### Runtime changes

When the client connects to the replication stream over the [Satellite protocol](../api/satellite.md), the system verifies that the local app and server have compatible migrations. New migrations streamed into Electric over logical replication are then streamed into the clients over the Satellite protocol and applied to the local SQLite database.
When the client connects to the replication stream over the [Satellite protocol](../api/satellite.md), the system verifies that the local app and server have compatible migrations. New migrations streamed into Electric over logical replication are then streamed into the clients over the Satellite protocol and applied to the local database.

This is done with [transactional causal consistency](./consistency.md). The DDL changes are marked as causal dependencies of any subsequent DML changes using that schema version, ensuring that the DDL changes are applied before any writes relying on them.

Expand All @@ -59,7 +59,7 @@ This is done with [transactional causal consistency](./consistency.md). The DDL

Data is synced onto local devices using [Shape subscriptions](../usage/data-access/shapes.md). Once data is synced into the local database, it can be queried using [static](../usage/data-access/queries.md#static-queries) and [live queries](../usage/data-access/queries.md#live-queries).

Local writes to SQLite are copied by triggers into the "oplog", a system table that keeps a log of pending write operations. In the local client app, the satellite process is in charge of replicating these operations over the [Satellite protocol](../api/satellite.md).
Local writes are copied by triggers into the "oplog", a system table that keeps a log of pending write operations. In the local client app, the satellite process is in charge of replicating these operations over the [Satellite protocol](../api/satellite.md).

[![Data flow diagramme](./_images/data-flow.png)](./_images/data-flow.jpg)

Expand All @@ -82,10 +82,10 @@ The satellite process then fires data change notifications, which triggers the [

### Local writes

Application code in the local app writes directly to the SQLite database using the [create, update and delete APIs](../usage/data-access/writes.md) from the type-safe database client. Before applying the writes, the client runs client-side validation to verify that the data is valid and will not be rejected once replicated.
Application code in the local app writes directly to the local SQLite or Postgres database using the [create, update and delete APIs](../usage/data-access/writes.md) from the type-safe database client. Before applying the writes, the client runs client-side validation to verify that the data is valid and will not be rejected once replicated.

:::caution
Direct writes to SQLite that don't use the client library will also be replicated by the satellite process. However, this skips validation, so is dangerous and best avoided.
Direct writes to the local database that don't use the client library will also be replicated by the satellite process. However, this skips validation, so is dangerous and best avoided.
:::

ElectricSQL aims to provide **finality** of local writes. That is to say: valid writes accepted locally should always be accepted by Electric and Postgres and never rejected unless invalid or unauthorised. This avoids having to code rollback handlers for local writes. This is key to simplifying local-first development.
Expand All @@ -95,7 +95,7 @@ Whilst local writes are final, they are still subject to concurrent merge semant

### Streaming into Electric

When the SQLite database migrations are generated from the Postgres DDL changes, triggers are added that automatically copy insert, update and delete operations on the tables to the "oplog" table. The satellite process then processes these operations by sending them to the Electric server over the Satellite protocol. Electric then applies server-side validation and authorisation before sending on to Postgres over the incoming logical-replication stream.
When the local database migrations are generated from the Postgres DDL changes, triggers are added that automatically copy insert, update and delete operations on the tables to the "oplog" table. The satellite process then processes these operations by sending them to the Electric server over the Satellite protocol. Electric then applies server-side validation and authorisation before sending on to Postgres over the incoming logical-replication stream.

:::note
Electric acts as a [logical replication publisher](https://www.postgresql.org/docs/current/logical-replication.html). This is why you configure a `LOGICAL_PUBLISHER_HOST` when deploying the Electric sync service -- so that Postgres can connect to consume inbound logical replication.
Expand Down
2 changes: 1 addition & 1 deletion docs/top-level-listings/integrations.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ import DocCardList from '@theme/DocCardList';

ElectricSQL is designed to work as a drop-in solution for existing applications and stacks.

It works with standard open-source Postgres and SQLite. In the frontend, you [adapt the standard SQLite driver](../integrations/drivers/index.md) for your target environment and bind live data to your existing [reactivity and component framework](../integrations/frontend/index.md).
It works with standard open-source Postgres and SQLite. In the frontend, you [adapt the standard SQLite, PGlite or Postgres driver](../integrations/drivers/index.md) for your target environment and bind live data to your existing [reactivity and component framework](../integrations/frontend/index.md).

In the backend, you can use your existing [web framework and migrations tooling](../integrations/backend/index.md) to manage content and evolve the database schema and standard [event sourcing tools](../integrations/event-sourcing/index.md) to integrate server-side workflows and background processing.

Expand Down
2 changes: 1 addition & 1 deletion docs/top-level-listings/reference.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ slug: /reference

import DocCardList from '@theme/DocCardList';

ElectricSQL is a conflict-free sync layer implementing active-active replication between Postgres in the cloud and SQLite on the local device. As such, it extends Postgres for [strong eventual consistency](../reference/consistency.md) and [optimal scale and resiliency](../reference/architecture.md).
ElectricSQL is a conflict-free sync layer implementing active-active replication between Postgres in the cloud and either SQLite or Postgres on the local device. As such, it extends Postgres for [strong eventual consistency](../reference/consistency.md) and [optimal scale and resiliency](../reference/architecture.md).

ElectricSQL is developed by a team of [experienced engineers and specialist distributed database experts](/about/team), under the guidance of:

Expand Down
4 changes: 2 additions & 2 deletions docs/usage/data-access/client.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,9 +21,9 @@ By default this outputs a `./src/generated/client` folder with an `index.ts` fil

## Instantiating the client

The exact code for instantiating your database client depends on the SQLite driver that you're using for your target environment. However, the steps are the same across drivers:
The exact code for instantiating your database client depends on the SQLite, PGlite or Postgres driver that you're using for your target environment. However, the steps are the same across drivers:

1. initialise an SQLite database connection (`conn`) using your underlying driver
1. initialise a database connection (`conn`) using your underlying driver
2. import your database schema (`schema`) from the `./generated/client` folder
3. optionally define a custom configuration `config`
4. pass these to your driver's `electrify` function to instantiate the client
Expand Down

0 comments on commit d5dda94

Please sign in to comment.