Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DBZ-2975: Prototype multiple offset contexts per task #15

Merged
merged 1 commit into from
Mar 4, 2021
Merged

DBZ-2975: Prototype multiple offset contexts per task #15

merged 1 commit into from
Mar 4, 2021

Conversation

morozov
Copy link

@morozov morozov commented Mar 2, 2021

Closes #12. The corresponding section of the design doc.

Change summary:

  1. OffsetContext.Loader is parametrized with <O extends OffsetContext> (see DBZ-2975: Extract offset context from object states to method signatures #13 for the reasoning).
  2. All messages produced by the SQL Server connector instead of having "server=serverName" as their source partition now have "server=serverName;database=databaseName" as their source partition. This is a breaking change from the PoV of existing users. An existing connector upgraded to this version won't be able to recover from the state produced by the previous version.

New APIs:

  1. TaskPartition interface. Describes the partition of the source to be processed by the task in terms of a given connector and provides its representation as a Kafka Connect source partition. For instance, an SqlServerTaskPartition describes a source database. Any other connector can implement this interface in its specific terms.
  2. TaskPartition.Provider interface. Provides a collection of task-specific partitions for a given task based on its configuration. E.g. a list SqlServerTaskPartitions each representing a database for a SQL Server connector task.
  3. TaskOffsetContext class. It's effectively a map of TaskPartition to OffsetContext describing the scope of work for a given task and its state. This map will be iterated over by different task components to process multiple databases.
  4. TaskOffsetContext.Loader class. Similarly to OffsetContext.Loader, loads the task offset context.

@morozov morozov marked this pull request as draft March 2, 2021 00:47
@morozov morozov linked an issue Mar 2, 2021 that may be closed by this pull request
@morozov morozov added the SQL Server Debezium connector for SQL Server label Mar 3, 2021
@morozov morozov marked this pull request as ready for review March 3, 2021 00:57
@morozov morozov merged commit be5bfde into sugarcrm:DBZ-2975 Mar 4, 2021
@morozov morozov deleted the issues/12 branch March 4, 2021 17:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
SQL Server Debezium connector for SQL Server
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Prototype API changes at the BaseSourceTask level
3 participants