fix: align skipIntervals to segmentGranularity in DataSourceCompactibleSegmentIterator - #20027
fix: align skipIntervals to segmentGranularity in DataSourceCompactibleSegmentIterator #20027cecemei wants to merge 4 commits into
Conversation
FrankChen021
left a comment
There was a problem hiding this comment.
| Severity | Findings |
|---|---|
| P0 | 0 |
| P1 | 0 |
| P2 | 1 |
| P3 | 1 |
| Total | 2 |
Reviewed 3 of 3 changed files.
Validation: git diff --check 1e1ee8cc454985197de5c6ceaf31a4d55e3ca39f..a5117d3ac085947f2dab1d2549408efaf9384573 passed. Builds and tests not run.
This is an automated review by Codex GPT-5.6-Luna(max)
|
Thanks for the changes, @cecemei ! I was under the impression that the code already ignores segments that cross the skip interval boundaries. The above snippet is from the method This method uses the But since the lookup interval has no overlap with any of the skip intervals (due to the @cecemei , did you encounter/write up a test case which currently fails due to misaligned granularities? |
The test case i added in this PR would fail without the change made. The case here is that we could have a skip interval in the middle of the day (10am - 12pm), and there're segments overlapping (0am - 10am) which get passed into findAndEnqueueSegmentsToCompact, so only the exact skip interval is not compacted but some segments during the day is still being compacted. |
Thanks for adding the test! The problem is that the |
kfaraz
left a comment
There was a problem hiding this comment.
Leaving minor suggestions to simplify the compaction skip reason.
| } | ||
|
|
||
| @Test | ||
| public void testSkipIntervalNotAlignedWithSegmentGranularityIsNotCompacted() |
There was a problem hiding this comment.
The test makes sense, thanks!
FrankChen021
left a comment
There was a problem hiding this comment.
| Severity | Findings |
|---|---|
| P0 | 0 |
| P1 | 0 |
| P2 | 1 |
| P3 | 0 |
| Total | 1 |
Reviewed 3 of 3 changed files.
Validation: git diff --check 1e1ee8cc454985197de5c6ceaf31a4d55e3ca39f4 97fc0a842e3152edc00eb6de3df6523442ac99d4 passed. Builds and tests were not run.
This is an automated review by Codex GPT-5.6-Luna(max)
| } | ||
| final DateTime alignedStart = segmentGranularity.bucketStart(interval.getStart()); | ||
| final DateTime endBucketStart = segmentGranularity.bucketStart(interval.getEnd()); | ||
| final DateTime alignedEnd = endBucketStart.isEqual(interval.getEnd()) |
There was a problem hiding this comment.
[P2] ETERNITY skip intervals overflow during alignment
If a configured or supplied skip interval is Intervals.ETERNITY and a segment granularity such as DAY is configured, bucketStart(MIN) moves the start before DateTimes.MIN and bucketEnd(MAX) moves the end after DateTimes.MAX. The resulting interval has a duration greater than Long.MAX_VALUE, so JodaUtils.condenseIntervals throws ArithmeticException from toDurationMillis() before compaction starts. Preserve ETERNITY or clamp aligned endpoints to the supported Druid bounds.
There was a problem hiding this comment.
Yeah, I guess we can add a sanity check in the constructor itself and if any of the skip intervals is eternity, we just skip compaction of all the segments.
Description
Follow-up to #20007, addressing #20007 (comment).
This PR has: