Enable Pinot realtime table ingestion without a deep store. Segment tar/zip and upload will thus be skipped.#10400
Enable Pinot realtime table ingestion without a deep store. Segment tar/zip and upload will thus be skipped.#10400chenboat wants to merge 1 commit intoapache:masterfrom
Conversation
|
|
||
| // True if and only if the segment upload to deep store is disabled. This is usefully for Pinot cluster which does not | ||
| // have any deep store. | ||
| private boolean _segmentUploadToDeepStoreDisabled; |
There was a problem hiding this comment.
we shall also disallow rebalance with downtime when this is disabled, right?
There was a problem hiding this comment.
Yes. If there is no deep store copy, the rebalance with downtime will result in segment data loss. I will add a note about this mode of ingestion with its limitation. We can also file a PR to detect the missing segments during rebalance.
| String.format("Caught exception while taring index directory from: %s to: %s", indexDir, segmentTarFile); | ||
| _segmentLogger.error(errorMessage, e); | ||
| _realtimeTableDataManager.addSegmentError(_segmentNameStr, new SegmentErrorInfo(now(), errorMessage, e)); | ||
| return null; |
There was a problem hiding this comment.
shall this return or continue (as if deep store was disabled)
There was a problem hiding this comment.
if deepstore is disabled, the protocol will continue because it still needs to send metadata file info about the segment to controller for segment completion.
There was a problem hiding this comment.
this behavior is the same as before and not changed (note the if condition is !deepStoreDisabled()). The idea then was then if segment tar/zip fails, the segment completion protocol just failed.
Jackie-Jiang
added
feature
release-notes
|
Currently we have multiple modes for LLC segment commit, shall we re-design the configs so that it is easier for users to understand?
|
|
To add to @Jackie-Jiang 's comment, right now for split-commit we have a "commitSegmentFile" step where we do a rename and some other PinotFS operations. Can we also discuss how this patch fits in with that? |
|
+1 to Jackie's comment above. |
5 participants
This PR enables a realtime table ingestion using Low Level Consumer (LLC) to run without deep store configured. The steps to tar/zip a segment and then upload will be skipped. The segment download will then be done through peer servers. A new table config "segmentUploadToDeepStoreDisabled" will be added to support this ingestion mode.
Instructions:
The PR has to be tagged with at least one of the following labels (*):
featureRemove these instructions before publishing the PR.
(**) Use
release-noteslabel for scenarios like: