Backport to 9x: Reduce duplication in taxonomy facets; always do counts #12966 #13358
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This is a large change, refactoring most of the taxonomy facets code and changing internal behaviour, without changing the API. There are specific API changes this sets us up to do later, e.g. retrieving counts from aggregation facets. 1. Move most of the responsibility from TaxonomyFacets implementations to TaxonomyFacets itself. This reduces code duplication and enables future development. Addresses genericity issue mentioned in apache#12553. 2. As a consequence, introduce sparse values to FloatTaxonomyFacets, which previously used dense values always. This issue is part of apache#12576. 3. Compute counts for all taxonomy facets always, which enables us to add an API to retrieve counts for association facets in the future. Addresses apache#11282. 4. As a consequence of having counts, we can check whether we encountered a label while faceting (count > 0), while previously we relied on the aggregation value to be positive. Closes apache#12585. 5. Introduce the idea of doing multiple aggregations in one go, with association facets doing the aggregation they were already doing, plus a count. We can extend to an arbitrary number of aggregations, as suggested in apache#12546. 6. Don't change the API. The only change in behaviour users should notice is the fix for non-positive aggregation values, which were previously discarded. 7. Add tests which were missing for sparse/dense values and non-positive aggregations.
|
Despite the "annoying" bits in the description, I don't expect this backport to be controversial, but reviews are welcome! I plan to wait over the weekend and then merge. |
stefanvodita
changed the title
stefanvodita
referenced
this pull request
May 13, 2024
…#13284) Our per-field vector and doc-values readers use `TreeMap`s but don't rely on the iteration order, so these `TreeMap`s can be replaced with more CPU/RAM-efficient `HashMap`s. The per-field postings reader stays on a `TreeMap` since it relies on the iteration order.
mikemccand
approved these changes
May 13, 2024
There was a problem hiding this comment.
Thanks @stefanvodita!
Make sure to update the main lucene/CHANGES.txt also to move the entry down to 9.11.0 section.
|
Thank you for the review, Mike! I'd already put the CHANGES entries in 9.11 tentatively, now they're correct 😄 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Annoying things I had to do to backport #12966:
TaxonomyFacetCountsstores the counts twice because it extendsIntTaxonomyFacets. The correct way would be to extendTaxonomyFacets, but I couldn't make that change, like I could forFastTaxonomyFacets, which was marked@lucene.experimental.useHashTableandrollupinTaxonomyFacetscompared tomain, from default to protected.TopOrdAndInt/FloatQueueas they were, since they are public API, and create new queues for our purposes,TopOrdAndInt/FloatNumberQueue, marked deprecated because they will go away in 10.x.