[Incremental Reprocessing] Multiplexed sync configs#651
Conversation
🦋 Changeset detectedLatest commit: 3276000 The changes in this PR will be included in the next version bump. This PR includes changesets to release 18 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
This comment was marked as resolved.
This comment was marked as resolved.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 72112de1fb
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
simolus3
left a comment
There was a problem hiding this comment.
These changes make sense to me after our offline discussion 👍 I have a few minor questions.
This refactors
HydratedSyncRulesto support multipleSyncConfiginstances. When multiple are passed in, we de-duplicate definitions. This currently requires the storage layer to assign unique ids/keys to each definition, whichHydratedSyncRulesthen uses for de-duplication. That process will be implemented in a future PR - nothing uses multiple sync configs yet.Why?
This will allow a replication stream/job to continue using
HydratedSyncConfigas-is to process multipleSyncConfigs at a time, provided that the storage implementation constructs it correctly.Further refactoring included
HydratedSyncRules->HydratedSyncConfig.keyinBucketDataScopeandParameterLookupScope.ParameterLookupDefinitionIdas a subset ofParameterLookupScope, excludingsourceand the newkey, to avoid the storage details leaking into the sync config parsing.AI usage
Used Codex gpt-5.5 for planning and implementation, manually guided and reviewed.