-
Notifications
You must be signed in to change notification settings - Fork 2.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[HUDI-7723] DayBasedCompactionStrategy supports io bounded #11169
base: master
Are you sure you want to change the base?
Conversation
@@ -60,6 +62,17 @@ public Comparator<String> getComparator() { | |||
return comparator; | |||
} | |||
|
|||
@Override | |||
public List<HoodieCompactionOperation> orderAndFilter(HoodieWriteConfig writeConfig, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
should we extend from BoundedIOCompactionStrategy
and by default does not limit the IO boundary(make it a maximum default value maybe)?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe add a boolean value is more reasonable. BoundedIOCompactionStrategy
has a default config hoodie.compaction.target.io
with value 512G. Following your idea,we should add a similar new config like hoodie.compaction.target.io
which make it maximum default value. That seems a little conflict.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
512G
itself is a value big enough for most of the production use cases, we can use it IMO.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's ok if 512G
is regarded as the maximum value. I fix it.
Change Logs
This pr is meant to:
Impact
no
Risk level (write none, low medium or high below)
low
Documentation Update
Describe any necessary documentation update if there is any new feature, config, or user-facing change. If not, put "none".
ticket number here and follow the instruction to make
changes to the website.
Contributor's checklist