Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow star-tree creation during segment load #5641

Merged
merged 1 commit into from
Jul 7, 2020

Conversation

Jackie-Jiang
Copy link
Contributor

@Jackie-Jiang Jackie-Jiang commented Jul 1, 2020

Description

Allow star-tree creation during segment load (and reload).
With this, star-tree can be added on the fly after updating the table config via reload.

NOTE:

  • This feature only applies to adding star-tree index config and generate star-trees to the existing segments (modifying/removing star-tree index config is not supported now)
  • Because the star-tree creation could consume a lot of system resources, added a boolean config enableDynamicStarTreeeCreation in IndexingConfig to enable/disable this feature (disabled by default).
  • This feature should only be enabled for testing purpose or in cluster without too much load on the servers and with relatively small segments and simple star-tree index config (servers need to have free system resources to create the star-tree).

Other changes:

  • Pass in enableDefaultStarTree into MultipleTreesBuilder instead of using null as a placeholder for default star-tree to enhance the readability
  • Make MultipleTreesBuilder closeable to close the segment properly

Release Notes

Introduced a new boolean config enableDynamicStarTreeeCreation in IndexingConfig to enable/disable star-tree creation during segment load.

@Jackie-Jiang Jackie-Jiang added the release-notes Referenced by PRs that need attention when compiling the next release notes label Jul 1, 2020
@mayankshriv
Copy link
Contributor

Would be good to include the default value of this config (hopefully it is false). Also, as a user, if I think StarTree will help, what is the guideline for me to choose building it outside Pinot vs at load time (how do I know whether my production servers won't die if I enable this feature)?

@Jackie-Jiang
Copy link
Contributor Author

@mayankshriv

  • Boolean member variable is always default to false (e.g. we don't have an explicit default value for aggregateMetrics, createInvertedIndexDuringSegmentGeneration etc.)
  • Star-tree is always created on the builder side. This feature is for users to add star-tree to the existing segments without star-tree. Added more descriptions in the PR.

@Jackie-Jiang
Copy link
Contributor Author

@mayankshriv Can you please take another look?

Copy link
Contributor

@mayankshriv mayankshriv left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please also update the docs to help users on choosing when to use this feature (specifically on how they can ensure that it won't put memory/cpu pressure on the server).

@Jackie-Jiang Jackie-Jiang force-pushed the star_tree_reload branch 2 times, most recently from e96bc95 to 3c2e73e Compare July 7, 2020 21:19
Allow star-tree creation during segment load (and reload).
With this, star-tree can be created for COMPLETED realtime segments.

Because the star-tree creation could consume a lot of system resources, added a boolean config 'enableDynamicStarTreeeCreation' in IndexingConfig to enable/disable this feature.

Other changes:
- Pass in `enableDefaultStarTree` into MultipleTreesBuilder instead of using null as a placeholder for default star-tree to enhance the readability
- Make MultipleTreesBuilder closeable to close the segment properly
@Jackie-Jiang Jackie-Jiang merged commit 903282c into apache:master Jul 7, 2020
@Jackie-Jiang Jackie-Jiang deleted the star_tree_reload branch July 7, 2020 22:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
release-notes Referenced by PRs that need attention when compiling the next release notes
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants