Add a new table config field for peer segment download. #5478
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
mcvsubbu
reviewed
Jun 1, 2020
There was a problem hiding this comment.
Please add validation code ("Must be http or https") in TableConfigUtils.validate() thanks
Also, add more documentation on the new config variable (at least a pointer to the design doc).
Please label the PR marking it for release notes.
Make sure you add documentation on the config in gitbook (after you merge this PR)
Done except that the last point about documentation. |
mcvsubbu
approved these changes
Jun 2, 2020
pinot-common/src/main/java/org/apache/pinot/common/utils/config/TableConfigUtils.java
Outdated
Show resolved
Hide resolved
pinot-common/src/main/java/org/apache/pinot/common/utils/config/TableConfigUtils.java
Outdated
Show resolved
Hide resolved
…g/TableConfigUtils.java Co-authored-by: Subbu Subramaniam <mcvsubbu@users.noreply.github.com>
…g/TableConfigUtils.java Co-authored-by: Subbu Subramaniam <mcvsubbu@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Add a new optional string field peerSegmentDownloadScheme to the SegmentsValidationAndRetentionConfig in the TableConfig. The value can be http or https.
The field will enable download of segments for both realtime and offline table segments from peer servers. In the beginning, only realtime table segments download will be supported. The design details can be found in this section of the cwiki doc.
@mcvsubbu
Upgrade Notes
Does this PR prevent a zero down-time upgrade? (Assume upgrade order: Controller, Broker, Server, Minion)
backward-incompat, and complete the section below on Release Notes)Does this PR fix a zero-downtime upgrade introduced earlier?
backward-incompat, and complete the section below on Release Notes)Does this PR otherwise need attention when creating release notes? Things to consider:
release-notesand complete the section on Release Notes)Release Notes
Add a new optional string field peerSegmentDownloadScheme to the SegmentsValidationAndRetentionConfig in the TableConfig. The value can be http or https.
The field will enable download of segments for both realtime and offline table segments from peer servers. Currently, only realtime table segment download during LLC segment completion protocol will be supported. For more details, please refer to the cwiki design doc "By-passing deepstore requirement for realtime segment completion".
Documentation
If you have introduced a new feature or configuration, please add it to the documentation as well.
See https://docs.pinot.apache.org/developers/developers-and-contributors/update-document