[WIP] [Postgres] stream while snapshotting

[rkistner]

Builds on #425.

This refactors Postgres replication to:

1. Split out snapshotting implementation from streaming replication (WalStream.ts).
2. Start streaming concurrently with the initial snapshot.

If a table needs a re-snapshot (e.g. due to replica identity changes), this also happens concurrently with streaming (although it still blocks the next commit until it completed). Truncating tables still block the replication stream.

This is not quite ready for general release yet - we need to consider the memory usage implications of doing snapshotting concurrently with streaming. We may need to make this an opt-in mode.

Background

Currently, the replication process is effectively linear / "single threaded". When a new sync rules version is deployed, we create a new replication stream, which performs a snapshot on each table, then starts streaming. This has a couple of limitations:

1. Replication is slower than it needs to be due to not being able to replicate tables concurrently.
2. After the initial table snapshots are complete, there could be a significant replication lag that we need to catch up on.

The changes here are also part of the bigger project to implement differential sync rule updates - only re-replicating for changed bucket definitions / sync stream definitions. Part of that requires switching to a single replication stream for all copies of sync rule versions, and this builds the base to implement that.

Implementation

To avoid consistency issues, we:

1. Ignore any rows picked up by streaming replication when we do a snapshot.
2. Implement soft-deletes to similarly prioritize deletes in the replication stream over rows in the table snapshot.

This is implemented in the storage layer in #425.

[changeset-bot]

⚠️ No Changeset found

Latest commit: 510173c

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR