Skip to content

fix OR filter partial index value matcher on cursor reset#18029

Merged
gianm merged 6 commits intoapache:masterfrom
clintropolis:fix-or-filter-partial-index-matcher-on-reset
May 22, 2025
Merged

fix OR filter partial index value matcher on cursor reset#18029
gianm merged 6 commits intoapache:masterfrom
clintropolis:fix-or-filter-partial-index-matcher-on-reset

Conversation

@clintropolis
Copy link
Member

@clintropolis clintropolis commented May 21, 2025

Description

Fixes a bug that can happen when a cursor is reset where the offset contains an OR filter bundle that contains 'partial index' value matchers (where the value matcher consists of one or more value matchers that are actually backed by iterators of the index bitmap). These index backed value matchers are stateful and assume that the offset will be advanced in order, and so must be reset if the underlying offset goes 'backwards' (e.g. from cursor reset).

The added tests fail without the changes in this PR, and also modified BaseFilterTest to have some cursor resets involved before collecting the values to try to prevent and help catch any other similar lurking issues.

Adding the cursor reset to the filter tests uncovered a flaw in RowBasedExpressionColumnValueSelector, where if a column had any numeric values, even if it previously had array values, then it would be no longer eligible to transform the expression into a map expression to handle mvds. Since this selector probably isn't used nearly as much anymore due to improved type information availability, i have just removed the 'ignored' marking functionality, so that we will always examine the input values to determine if they are scalar or not.


This PR has:

  • been self-reviewed.
  • added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
  • added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
  • been tested in a test Druid cluster.

…ColumnValueSelector by removing ignored marking since it could incorrectly eliminate columns that had some mvd rows and some number rows (uncommon, but im not sure it is technically illegal)
@gianm gianm merged commit 9bb25e6 into apache:master May 22, 2025
74 checks passed
@clintropolis clintropolis deleted the fix-or-filter-partial-index-matcher-on-reset branch May 22, 2025 20:01
@capistrant capistrant added this to the 34.0.0 milestone Jul 22, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants