Skip unknown index types in SingleFileIndexDirectory#17992
Skip unknown index types in SingleFileIndexDirectory#17992xiangfu0 merged 1 commit intoapache:masterfrom
Conversation
Gracefully handle unknown index types in the index map by logging a warning and skipping the entry instead of throwing an exception. This improves forward compatibility when segments contain index types not yet recognized by the current version. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Improves forward compatibility of segment loading by preventing SingleFileIndexDirectory from failing when encountering index map entries for index types not recognized by the running Pinot version.
Changes:
- Catch
IllegalArgumentExceptionfromIndexKey.fromIndexName()during index map load. - Log a warning and skip unknown index entries instead of throwing and aborting load.
| } catch (IllegalArgumentException e) { | ||
| LOGGER.warn("Skipping unknown index type: {} for column: {} in segment: {}", parsedKeys[1], parsedKeys[0], | ||
| _segmentDirectory); | ||
| continue; |
There was a problem hiding this comment.
This warning will be emitted once per map entry. Since each index typically has at least START_OFFSET and SIZE entries, an unknown index type will likely log the same warning multiple times (log spam). Consider deduplicating (e.g., keep a Set of (column,indexType) already-warned) or lowering to DEBUG after the first occurrence.
| IndexKey indexKey; | ||
| try { | ||
| indexKey = IndexKey.fromIndexName(parsedKeys[0], parsedKeys[1]); | ||
| } catch (IllegalArgumentException e) { | ||
| LOGGER.warn("Skipping unknown index type: {} for column: {} in segment: {}", parsedKeys[1], parsedKeys[0], | ||
| _segmentDirectory); | ||
| continue; | ||
| } |
There was a problem hiding this comment.
This change alters load behavior to skip unknown index types instead of failing. Please add a unit test that writes an index map entry with an unrecognized index id and asserts that SingleFileIndexDirectory still loads (and that known indices remain accessible).
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #17992 +/- ##
=============================================
+ Coverage 34.20% 63.26% +29.06%
- Complexity 795 1541 +746
=============================================
Files 3198 3200 +2
Lines 193998 194042 +44
Branches 29865 29870 +5
=============================================
+ Hits 66348 122767 +56419
+ Misses 122065 61632 -60433
- Partials 5585 9643 +4058
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
Summary
SingleFileIndexDirectoryby catchingIllegalArgumentExceptionfromIndexKey.fromIndexName()Test plan
pinot-segment-localmodule:./mvnw -pl pinot-segment-local -am verifySingleFileIndexDirectoryTestpasses🤖 Generated with Claude Code