Repository navigation
concurrent mysql read - #60218
Merged
Rodi Reich Zilberman (rodireich) merged 21 commits intoMay 29, 2025
Merged
concurrent mysql read#60218
Conversation
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎
|
Contributor
|
Rodi Reich Zilberman (rodireich)
marked this pull request as ready for review
May 13, 2025 17:10
…-mysql-partitions-with-numeric-key
Contributor
|
Note Detected that there are differences in the Gradle dependencies. |
Yue Li (theyueli)
approved these changes
May 29, 2025
Rodi Reich Zilberman (rodireich)
deleted the
12727-concurrent-mysql-implement-split-for-mysql-partitions-with-numeric-key
branch
May 29, 2025 20:38
Matt Bayley (mwbayley)
left a comment
Matt Bayley (mwbayley)
left a comment
Contributor
There was a problem hiding this comment.
I was mid-review when I noticed this got merged. Better late than never?
| * Interface to allow each toolkit to return the active [PartitionsCreatorFactory] class For | ||
| * [ReadOperation] to be able to iterator on active factories only. | ||
| */ | ||
| interface PartitionsCreatorFactorySupplier<T : PartitionsCreatorFactory> : Supplier<T> |
Contributor
There was a problem hiding this comment.
Do we need this interface? Can we just use Supplier<T: PartitionsCreatorFactory> directly?
|
|
||
| /** | ||
| * Interface to allow each toolkit to return the active [PartitionsCreatorFactory] class For | ||
| * [ReadOperation] to be able to iterator on active factories only. |
Contributor
There was a problem hiding this comment.
sp: iterate
| core = 'extract' | ||
| toolkits = ['extract-jdbc', 'extract-cdc'] | ||
| cdk = '0.409' | ||
| cdk = 'local' |
Contributor
There was a problem hiding this comment.
we're trying to avoid doing this these days, right?
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
In preparation for the speed increase, mysql has to be to be able to read a table using concurrent partitions.
This change turns mysql to concurrent.
How
Because the mysql server doesn't have the necessary
SELECT … FROM TABLE SAMPLE, we need to build partitions boundaries using a different algorithm.This change applies math in order implement boundaries.
It also plugs into the existing framework of partitions factory and implements the necessary Splittable partitions variants for all relevant sync types (full-refresh, CDC, cursor) and phases (snapshot, incremental)
If you have a table with a highest primary key
500and the table is partitioned into 5 partitions the boundaries will be set as0…10
10…20
20…30
30…40
40…50
Review guide
MySqlSourceJdbcPartition.kt: DI overload of partitions creator. routing code to mysql' own implementation.MySqlSourceJdbcPartitionFactory: the new implementation ofsplit()User Impact
The intention is to merge this code initially turned off by leaving application.yml with
mode: sequential.While we keep testing an validate with boosted mode additions.
Can this PR be safely reverted and rolled back?