Configurable storage versions

[rkistner]

This adds config.storage_version, which can currently be 2 or 3.

For background, see:

• Storage version #487

In most cases, the storage version should not be configured explicitly - the latest stable version will be used automatically. This adds a config.storage_version property to cater for some specific cases:

1. Downgrade: Say the current storage version is 4, but you want to downgrade to an older service version that only support up to version 2. To do that, first deploy with config.storage_version: 2 in the sync config, wait for reprocessing to complete, then downgrade the service.
2. Experimental versions: Allows testing of storage changes in [MongoDB Storage] Storage v3 for incremental reprocessing #585.
3. Holding back on an upgrade. In some cases, you may want to make sync config changes but stay on an older storage version.

Version strategy

Together with this, I propose a storage version strategy:

1. Even numbers are always stable: Once support for an even number is added, future versions should continue supporting that storage version as-is, until officially deprecated.
2. Odd numbers are explicitly unstable: The underlying format may change at any time, and support for the version may be removed at any time. It should only be used for testing, never for production.

So right now we have version 2 as stable, and version 3 as unstable. Once all changes in version 3 has been finalized, we can stabilize the format as version 4, and remove support for version 3.

This strategy is not explicitly captured by the code yet - other than specifically marking version 2 as stable and 3 as unstable.

Version 1 is a special legacy case: This is used for all instances where the version was not defined. We support using storage version 1, but the user cannot configure sync config for version 1. We could consider doing the same for other deprecated storage versions in the future.

Implementation

The supported storage versions are now kinda duplicated between sync-rules and service-core packages. It does need to be available in sync-rules package, since we need to be able to validate the sync config without having access to service-core.

We could consider merging the service-core functionality into that.
This is now split in a way that the sync-rules package purely does validation and parsing a version number, while the service-core package assigns more specific behavior.

[changeset-bot]

🦋 Changeset detected

Latest commit: 509cd6c

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 18 packages

Name	Type
@powersync/service-module-postgres-storage	Patch
@powersync/service-module-mongodb-storage	Patch
@powersync/service-core-tests	Patch
@powersync/service-module-postgres	Patch
@powersync/service-core	Patch
@powersync/service-sync-rules	Patch
@powersync/lib-service-mongodb	Patch
@powersync/service-schema	Patch
@powersync/service-module-mongodb	Patch
@powersync/service-module-mssql	Patch
@powersync/service-module-mysql	Patch
@powersync/service-image	Patch
@powersync/service-module-core	Patch
test-client	Patch
@powersync/service-jpgwire	Patch
@powersync/lib-services-framework	Patch
@powersync/lib-service-postgres	Patch
@powersync/service-rsocket-router	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[simolus3]

Using this to downgrade storage versions is clever, I also like the approach of odd/even numbers for stability 👍

I've added the product-visibility label because we probably want to document this when it gets released.

The base branch was changed.