Improve the realtime time creation unit test #6032
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
snleee
force-pushed
the
improve-realtime-table-creation
branch
from
91e1658
to
295371e
Compare
Jackie-Jiang
approved these changes
Sep 17, 2020
|
@Jackie-Jiang I will rename the description and title. |
1. Added the precondition check for time interval from segment metadata 2. Added the unit test to check the case when the invalid time column name is configured in the table config
snleee
force-pushed
the
improve-realtime-table-creation
branch
from
295371e
to
cb5e34e
Compare
snleee
changed the title
mcvsubbu
reviewed
Sep 18, 2020
| @@ -504,6 +504,8 @@ private LLCRealtimeSegmentZKMetadata updateCommittingSegmentZKMetadata(String re | |||
| committingSegmentZKMetadata.setDownloadUrl(isPeerURL(committingSegmentDescriptor.getSegmentLocation()) | |||
| ? CommonConstants.Segment.METADATA_URI_FOR_PEER_DOWNLOAD : committingSegmentDescriptor.getSegmentLocation()); | |||
| committingSegmentZKMetadata.setCrc(Long.valueOf(segmentMetadata.getCrc())); | |||
| Preconditions.checkNotNull(segmentMetadata.getTimeInterval(), | |||
There was a problem hiding this comment.
Consumption will stop if this condition hits. Are we sure that is what we want?
There was a problem hiding this comment.
@mcvsubbu This condition should not happen after the fix from #5966. When this happens, the realtime segment commit would anyway fail and the consumption would be stopped due to the null pointer exception. (egmentMetadata.getTimeInterval().getStartMillis() gets called at the following line). I'm just trying to add more information to the logs here since the log message was not informative when we hit that.
We recently saw the null pointer error issue (and the consumption actually stopped) when the time column is misconfigured.
2020/09/17 20:36:52.666 ERROR [SegmentCompletionFSM_XXXXX__3__0__20200915T2142Z] [grizzly-http-server-3] [pinot-controller] [] Caught exception while committing segment metadata for segment: XXXXX__3__0__20200915T2142Z
java.lang.NullPointerException: null
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
is configured in the table config