Add help methods to check if segment needs reprocessing#7925
Add help methods to check if segment needs reprocessing#7925Jackie-Jiang merged 3 commits intoapache:masterfrom
Conversation
…table config and schema
|
Found I was not able to reopen the previously closed PR: #7898 after rebasing new master. So I opened this PR, which doesn't add new changes but address comments from @Jackie-Jiang @siddharthteotia: Sorry about the inconvenience. Please continue the review here. |
Codecov Report
@@ Coverage Diff @@
## master #7925 +/- ##
=============================================
- Coverage 71.15% 27.67% -43.48%
=============================================
Files 1593 1584 -9
Lines 82365 82109 -256
Branches 12270 12260 -10
=============================================
- Hits 58609 22726 -35883
- Misses 19806 57320 +37514
+ Partials 3950 2063 -1887
Flags with carried forward coverage won't be shown. Click here to find out more.
Continue to review full report at Codecov.
|
26e58b3 to
9b814b2
Compare
9b814b2 to
f83f88f
Compare
...java/org/apache/pinot/segment/local/segment/index/loader/bloomfilter/BloomFilterHandler.java
Outdated
Show resolved
Hide resolved
4f12f94 to
21debfe
Compare
Description
Add helper methods to check if a segment needs reprocessing by comparing its metadata with the table config and schema.
Right now, the checking and reprocessing logic are mixed together in ImmutableSegmentLoader.load() method. The helper methods added here will allow us to check consistency before conducting any real reprocessing that can be heavy. For example, when a segment doesn't need reprocessing, we can save all the upfront cost of preparing the data for reprocessing, like downloading some data from remote tier backend.
This PR hasn't changed the existing control flow yet. The next one will.
Upgrade Notes
Does this PR prevent a zero down-time upgrade? (Assume upgrade order: Controller, Broker, Server, Minion)
backward-incompat, and complete the section below on Release Notes)Does this PR fix a zero-downtime upgrade introduced earlier?
backward-incompat, and complete the section below on Release Notes)Does this PR otherwise need attention when creating release notes? Things to consider:
release-notesand complete the section on Release Notes)Release Notes
Documentation