Record if block API has been used in SegmentInfo #12685
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
If the add/updateDocuments(List<>) API is used, lucene guarantees that all documents are indexed in the same segment with consecutive document IDs. This enables features like nested documents etc. This change records the usage of this API in SegmentsInfo and preserves this property across merges. Relates to apache#12665
s1monw
commented
Oct 16, 2023
5 tasks
s1monw
commented
Oct 16, 2023
| if (docCount < 0) { | ||
| throw new CorruptIndexException("invalid docCount: " + docCount, input); | ||
| } | ||
| final boolean isCompoundFile = input.readByte() == SegmentInfo.YES; |
There was a problem hiding this comment.
Instead of writing a byte each time here and change index format we could also write a bitset to mark features like this. it might be easier down the road. not sure how often it happens.
s1monw
changed the title
jpountz
reviewed
Oct 16, 2023
lucene/core/src/test/org/apache/lucene/index/TestAddIndexes.java
Outdated
Show resolved
Hide resolved
|
@jpountz I pushed new commits, wanna take a new look |
s1monw
added a commit
that referenced
this pull request
Oct 23, 2023
If the add/updateDocuments(List<>) API is used, lucene guarantees that all documents are indexed in the same segment with consecutive document IDs. This enables features like nested documents etc. This change records the usage of this API in SegmentsInfo and preserves this property across merges. Relates to #12665
|
FYI we've seen failures on TestIndexWriter recently, which are reproducible (e.g. https://ci-builds.apache.org/job/Lucene/job/Lucene-NightlyTests-9.x/720/). I ran git bisect and it pointed to this change. I'm unsure why though. |
s1monw
added a commit
that referenced
this pull request
Oct 25, 2023
s1monw
added a commit
that referenced
this pull request
Oct 25, 2023
|
I pushed fixes... thanks @jpountz |
javanna
added a commit
to elastic/elasticsearch
that referenced
this pull request
Oct 26, 2023
This is needed following apache/lucene#12685 and apache/lucene#12582
javanna
added a commit
to elastic/elasticsearch
that referenced
this pull request
Oct 26, 2023
…foFormat These are needed after apache/lucene#12685
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
If the add/updateDocuments(List<>) API is used, lucene guarantees that all documents are indexed in the same segment with consecutive document IDs. This enables features like nested documents etc. This change records the usage of this API in SegmentInfo and preserves this property across merges.
Relates to #12665