New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Improve compaction tutorial to demonstrate compaction with keepSegmentGranularity = true #7079
Conversation
…tGranularity = true
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM, nice example 👍
@@ -35,7 +36,7 @@ It will also be helpful to have finished [Tutorial: Loading a file](../tutorials | |||
|
|||
## Load the initial data | |||
|
|||
For this tutorial, we'll be using the Wikipedia edits sample data, with an ingestion task spec that will create a separate segment for each hour in the input data. | |||
For this tutorial, we'll be using the Wikipedia edits sample data, with an ingestion task spec that will create 1-3 segments per hour in the input data. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
👍 for the link to segment sizing optimization above, but it would maybe also be worth mentioning that these settings are artificially constrained to produce extra segments so that the example data can be used for the compaction example. I don't think it's required though, so up to you if you modify this section.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think it makes sense. Will add.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Added. Thanks!
Should the tutorial also link to the docs on setting up automatic compaction? |
…tGranularity = true (apache#7079) * Improve compaction tutorial to demonstrate compaction with keepSegmentGranularity = true * typo * add a warning
The current tutorial demonstrates the compaction with
keepSegmentGranularity
= false, but it's not recommended and a deprecated behavior. This PR changes it to whenkeepSegmentGranularity
= true.