fix: Allow bootstrap to complete without updating col stats#12952
Open
CTTY wants to merge 5 commits intoapache:masterfrom
Open
fix: Allow bootstrap to complete without updating col stats#12952CTTY wants to merge 5 commits intoapache:masterfrom
CTTY wants to merge 5 commits intoapache:masterfrom
Conversation
yihua
reviewed
Mar 12, 2025
...lient-common/src/main/java/org/apache/hudi/table/action/commit/BaseCommitActionExecutor.java
Outdated
Show resolved
Hide resolved
| updateColumnsToIndexForColumnStats(metaClient, columnsToIndex); | ||
| return null; | ||
| }); | ||
| } catch (UnsupportedOperationException uoe) { |
Contributor
There was a problem hiding this comment.
Could we catch a specific exception for bootstrap here?
Contributor
There was a problem hiding this comment.
Have you checked that there is no failed deltacommit in MDT this case and files partition is intact, and the index ready to use does not contain col_stats in the table config?
Contributor
There was a problem hiding this comment.
Would be good to add this validation in the test.
yihua
reviewed
Mar 12, 2025
| return null; | ||
| }); | ||
| } catch (UnsupportedOperationException uoe) { | ||
| LOG.warn("Failed to update col stats, bootstrap doesn't support col stats", uoe); |
Contributor
There was a problem hiding this comment.
Similarly, let’s add or modify a test case to increase branch coverage.
CTTY
force-pushed
the
ctty/fix-ba-colstats
branch
from
315f643 to
cd644f1
Compare
Contributor
Author
|
#12977 seems to be able to fix this issue, I have not tested it yet. I'll test it later |
Contributor
Author
|
This issue exists on the released Hudi 1.0.2 as well and will be needed |
…able/action/commit/BaseCommitActionExecutor.java Co-authored-by: Y Ethan Guo <ethan.guoyihua@gmail.com>
CTTY
force-pushed
the
ctty/fix-ba-colstats
branch
from
d29bc45 to
74e0ae3
Compare
CTTY
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Describe the issue this Pull Request addresses
Since #12529 ,
BaseCommitActionExecutorwould update col stats by default. But bootstrap operation doesn't support col stats and will fail the bootstrapinghudi/hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/table/action/bootstrap/SparkBootstrapCommitActionExecutor.java
Line 225 in 63362ad
This PR is to allow bootstrap to complete without updating col stats.
Summary and Changelog
Impact
None
Risk Level
None
Documentation Update
None
Contributor's checklist