[HUDI-6329] Adjust the partitioner automatically for flink consistent hashing index #9087
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…uld detect whether clustering service occurs and automatically adjust the partitioner and write function if clustering service happens.
beyond1920
force-pushed
the
flink-chi-3task
branch
from
1bc4ea7
to
92e8459
Compare
danny0405
reviewed
Jul 1, 2023
hudi-flink-datasource/hudi-flink/src/test/java/org/apache/hudi/sink/utils/MockCollector.java
Outdated
Show resolved
Hide resolved
danny0405
reviewed
Jul 1, 2023
| HoodieFlinkTable table = writeClient.getHoodieTable(); | ||
| Option<HoodieInstant> latestPendingReplaceInstant = table.getActiveTimeline().filterPendingReplaceTimeline().lastInstant(); | ||
| if (latestPendingReplaceInstant.isPresent() && latestPendingReplaceInstant.get().getTimestamp().compareTo(lastRefreshInstant) > 0) { | ||
| LOG.info("Found new pending replacement commit. Last pending replacement commit is {}.", latestPendingReplaceInstant); |
There was a problem hiding this comment.
In order to make sure this is a CLUSTERING instant, you may also needs to check the plan, because clustering and insert overwrite share the same kind of instant type: REPLACE_COMMIT. There is a util method can do this: ClusteringUtil#isClusteringInstant
There was a problem hiding this comment.
+1. I could add a check here, using ClusteringUtils#getPendingClusteringInstantTimes to filter out pending clustering instants.
danny0405
reviewed
Jul 1, 2023
...flink-datasource/hudi-flink/src/main/java/org/apache/hudi/configuration/OptionsResolver.java
Outdated
Show resolved
Hide resolved
… bucket index 2. Check replacement is clustering
beyond1920
force-pushed
the
flink-chi-3task
branch
from
47e33ac
to
cc9a853
Compare
danny0405
changed the title
|
6329.patch.zip |
danny0405
reviewed
Jul 3, 2023
| private List<String> indexKeyFields; | ||
| private Map<String, Pair<String, ConsistentBucketIdentifier>> partitionToIdentifier; | ||
| private String lastRefreshInstant = HoodieTimeline.INIT_INSTANT_TS; | ||
|
|
There was a problem hiding this comment.
I didn't see lastRefreshInstant is used anywhere.
There was a problem hiding this comment.
Sorry to miss that, it is used to check whether there is new pending clustering request.
danny0405
changed the title
danny0405
approved these changes
Jul 4, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Change Logs
This pr is the third subtask of HUDI-4373.
It focuses on resolving the resizing cases in the write pipelines. It would detect whether clustering service occurs and automatically adjust the partitioner and write function if clustering service happens.
ps: This work is follow up of #6737. Thanks for contribution @YuweiXiao
Impact
NA
Risk level (write none, low medium or high below)
NA
Documentation Update
All documents update would be introduced in the final subtask of HUDI-4373.
Contributor's checklist