-
Notifications
You must be signed in to change notification settings - Fork 1.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Allow star-tree creation during segment load #5641
Conversation
Would be good to include the |
|
@mayankshriv Can you please take another look? |
pinot-core/src/main/java/org/apache/pinot/core/segment/index/loader/IndexLoadingConfig.java
Outdated
Show resolved
Hide resolved
pinot-spi/src/main/java/org/apache/pinot/spi/config/table/IndexingConfig.java
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please also update the docs to help users on choosing when to use this feature (specifically on how they can ensure that it won't put memory/cpu pressure on the server).
e96bc95
to
3c2e73e
Compare
Allow star-tree creation during segment load (and reload). With this, star-tree can be created for COMPLETED realtime segments. Because the star-tree creation could consume a lot of system resources, added a boolean config 'enableDynamicStarTreeeCreation' in IndexingConfig to enable/disable this feature. Other changes: - Pass in `enableDefaultStarTree` into MultipleTreesBuilder instead of using null as a placeholder for default star-tree to enhance the readability - Make MultipleTreesBuilder closeable to close the segment properly
3c2e73e
to
6172078
Compare
Description
Allow star-tree creation during segment load (and reload).
With this, star-tree can be added on the fly after updating the table config via reload.
NOTE:
enableDynamicStarTreeeCreation
inIndexingConfig
to enable/disable this feature (disabled by default).Other changes:
enableDefaultStarTree
into MultipleTreesBuilder instead of using null as a placeholder for default star-tree to enhance the readabilityRelease Notes
Introduced a new boolean config
enableDynamicStarTreeeCreation
inIndexingConfig
to enable/disable star-tree creation during segment load.