Skip to content

[Incremental Reprocessing] Multiplexed sync configs#651

Merged
rkistner merged 16 commits into
mainfrom
multi-processor
May 29, 2026
Merged

[Incremental Reprocessing] Multiplexed sync configs#651
rkistner merged 16 commits into
mainfrom
multi-processor

Conversation

@rkistner
Copy link
Copy Markdown
Contributor

This refactors HydratedSyncRules to support multiple SyncConfig instances. When multiple are passed in, we de-duplicate definitions. This currently requires the storage layer to assign unique ids/keys to each definition, which HydratedSyncRules then uses for de-duplication. That process will be implemented in a future PR - nothing uses multiple sync configs yet.

Why?

This will allow a replication stream/job to continue using HydratedSyncConfig as-is to process multiple SyncConfigs at a time, provided that the storage implementation constructs it correctly.

Further refactoring included

  • Rename HydratedSyncRules -> HydratedSyncConfig.
  • Include key in BucketDataScope and ParameterLookupScope.
  • Split out ParameterLookupDefinitionId as a subset of ParameterLookupScope, excluding source and the new key, to avoid the storage details leaking into the sync config parsing.

AI usage

Used Codex gpt-5.5 for planning and implementation, manually guided and reviewed.

@changeset-bot
Copy link
Copy Markdown

changeset-bot Bot commented May 26, 2026

🦋 Changeset detected

Latest commit: 3276000

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 18 packages
Name Type
@powersync/service-sync-rules Minor
@powersync/service-module-postgres-storage Patch
@powersync/service-module-mongodb-storage Patch
@powersync/service-module-postgres Patch
@powersync/service-module-mongodb Patch
@powersync/service-core Patch
@powersync/service-module-mssql Patch
@powersync/service-module-mysql Patch
@powersync/service-jpgwire Patch
@powersync/service-core-tests Patch
@powersync/lib-services-framework Patch
@powersync/service-schema Patch
@powersync/service-image Patch
@powersync/service-module-core Patch
test-client Patch
@powersync/lib-service-postgres Patch
@powersync/service-rsocket-router Patch
@powersync/lib-service-mongodb Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@rkistner rkistner marked this pull request as ready for review May 26, 2026 09:43
@rkistner rkistner requested a review from simolus3 May 26, 2026 09:43
@rkistner

This comment was marked as resolved.

chatgpt-codex-connector[bot]

This comment was marked as resolved.

Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 72112de1fb

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread packages/sync-rules/src/HydratedSyncConfig.ts Outdated
Comment thread packages/sync-rules/src/HydratedSyncConfig.ts
Copy link
Copy Markdown
Contributor

@simolus3 simolus3 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These changes make sense to me after our offline discussion 👍 I have a few minor questions.

Comment thread packages/service-core/src/storage/PersistedSyncRulesContent.ts Outdated
Comment thread packages/sync-rules/src/HydratedSyncConfig.ts Outdated
Comment thread packages/sync-rules/src/HydrationState.ts Outdated
Comment thread packages/sync-rules/src/HydrationState.ts Outdated
chatgpt-codex-connector[bot]

This comment was marked as resolved.

@rkistner rkistner requested a review from simolus3 May 29, 2026 09:14
chatgpt-codex-connector[bot]

This comment was marked as resolved.

@rkistner rkistner merged commit 6e2a57e into main May 29, 2026
44 checks passed
@rkistner rkistner deleted the multi-processor branch May 29, 2026 14:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants