Upgrade to Lucene 9.9.0 #102782
Merged
Upgrade to Lucene 9.9.0 #102782
Commits on Sep 20, 2023
-
Adds new max_inner_product vector similarity function (elastic#99527)
Adds new max_inner_product vector similarity function. This differs from dot_product in the following ways: Doesn't require vectors to be normalized Scales the similarity between vectors differently to prevent negative scores
Commits on Sep 21, 2023
Commits on Sep 22, 2023
-
-
Refactor changes to IndexVersion. (elastic#99312)
This adds a version for the Lucene upgrade and adjusts some tests.
Commits on Sep 23, 2023
Commits on Sep 24, 2023
Commits on Sep 25, 2023
Commits on Sep 26, 2023
Commits on Sep 27, 2023
-
Add RandomAccessInput#length to SeekTrackingDirectoryWrapper (elastic…
…#99962) Fix compiling error after adding a new method to RandomAccessInput interface.
Commits on Sep 28, 2023
Commits on Sep 29, 2023
Commits on Sep 30, 2023
Commits on Oct 1, 2023
Commits on Oct 2, 2023
Commits on Oct 3, 2023
Commits on Oct 5, 2023
Commits on Oct 6, 2023
Commits on Oct 7, 2023
Commits on Oct 8, 2023
Commits on Oct 9, 2023
Commits on Oct 10, 2023
Commits on Oct 11, 2023
Commits on Oct 12, 2023
Commits on Oct 13, 2023
Commits on Oct 14, 2023
Commits on Oct 15, 2023
Commits on Oct 16, 2023
Commits on Oct 17, 2023
Commits on Oct 18, 2023
Commits on Oct 19, 2023
Commits on Oct 20, 2023
Commits on Oct 21, 2023
Commits on Oct 22, 2023
Commits on Oct 23, 2023
Commits on Oct 24, 2023
Commits on Oct 25, 2023
Commits on Oct 26, 2023
-
Update references to Lucene95Codec
This is needed following apache/lucene#12685 and apache/lucene#12582
-
-
-
Fix compile errors on Lucene62SegmentInfoFormat and Lucene50SegmentIn…
…foFormat These are needed after apache/lucene#12685
Commits on Oct 27, 2023
Commits on Oct 28, 2023
Commits on Oct 29, 2023
Commits on Oct 30, 2023
Commits on Oct 31, 2023
-
Switch ContextIndexSearcher to use Lucene's TaskExecutor (elastic#101537
) We have contributed back to Lucene the changes we had made around running concurrent tasks. These include waiting for all tasks to finish when an exception is thrown, as well as not starting tasks when one of the previously run tasks throws an exception. The execution of concurrent tasks is now generalized within Lucene and exposed through a TaskExecutor that can be retrieved from the IndexSearcher and used to run tasks. We still have customizations that require us to override some of the search method, but with this change we rely on standard Lucene's behaviour for running concurrent tasks.
Commits on Nov 1, 2023
Commits on Nov 2, 2023
Commits on Nov 3, 2023
Commits on Nov 4, 2023
Commits on Nov 5, 2023
Commits on Nov 6, 2023
Commits on Nov 7, 2023
Commits on Nov 8, 2023
Commits on Nov 9, 2023
-
Fix compile errors (elastic#101874)
IndexDiskUsageAnalyzer and IndexDiskUsageAnalyzerTests, as well as CompletionFieldMapper, CompletionFieldMapperTests and CompletionStatsCacheTests need adjusting after apache/lucene#12741 , to refer to the latest postings format. KuromojiTokenizerFactory needs adjusting after apache/lucene#12390
-
Adjust SortField comparators to use new Pruning API (elastic#101983)
Introduced in apache/lucene#12405 We should account for the changes in our overrides and API. Now, to indicate that no skipping can occur, we utilize `Pruning.NONE`.
Commits on Nov 10, 2023
Commits on Nov 11, 2023
Commits on Nov 12, 2023
Commits on Nov 13, 2023
-
Resolve compile error in DenseVectorFieldMapper (elastic#102066)
* Resolve compile error in DenseVectorFieldMapper A change in Lucene99HnswVectorsFormat requires that we adapt our code, see apache/lucene#12729 * Add new Lucene file extensions * Fixing format name check --------- Co-authored-by: Benjamin Trent <4357155+benwtrent@users.noreply.github.com>
Commits on Nov 14, 2023
Commits on Nov 15, 2023
-
Fix failing tests due to new query to string format (elastic#102186)
There have been some changes around range query toString format. This commit adjusts tests expecting particular outputs from the previous Lucene version
Commits on Nov 16, 2023
Commits on Nov 17, 2023
Commits on Nov 18, 2023
Commits on Nov 19, 2023
Commits on Nov 20, 2023
Commits on Nov 21, 2023
Commits on Nov 22, 2023
Commits on Nov 23, 2023
Commits on Nov 24, 2023
Commits on Nov 25, 2023
Commits on Nov 26, 2023
Commits on Nov 27, 2023
Commits on Nov 28, 2023
Commits on Nov 29, 2023
-
-
Add byte quantization for float vectors in HNSW (elastic#102093)
Adds new `quantization_options` to `dense_vector`. This allows for vectors to be automatically quantized to `byte` when indexed. Example: ``` PUT vectors { "mappings": { "properties": { "my_vector": { "type": "dense_vector", "index": true, "index_options": { "type": "int8_hnsw" } } } } } ``` When querying, the query vector is automatically quantized and used when querying the HNSW graph. This reduces the memory required to only `25%` of what was previously required for `float` vectors at a slight loss of accuracy. This is currently only available when `index: true` and when using `hnsw` -
-
-
Commits on Nov 30, 2023
-
-
-
-
Fixing byte quantized search test flakiness
Quantized search result scores can be slightly different depending on if they are merged into a single segment, etc.
-
-
-
Commits on Dec 1, 2023
Commits on Dec 2, 2023
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.