[Refactor](inverted index) refactor inverted index compound predicates evaluate logic #38908 #41385

airborne12 · 2024-09-27T03:30:15Z

cherry pick from #38908

…s evaluate logic (apache#38908) This PR addresses several key issues related to the compound condition support in the inverted index, and optimization for index skipping without returning to the table: 1. **Unified Handling of `expr` and `column predicate`**: - Combined the processing of inverted index-related `column predicate` and `expr`. - Ensured that compound conditions involving both `column predicate` and `expr` are processed uniformly to reduce complexity and improve robustness. 2. **Optimized the Execution of Compound Conditions**: - Removed the logic in `scan_operator` that normalized compound predicates by pushing down logic to `_common_expr_ctxs_push_down` where `expr` contexts are managed. - Added `evaluate_inverted_index` support to the `vexpr` and function layers, such as `function comparison` and `function collection_in`. - Introduced new data structures in `VExprContext` to store results from `evaluate_inverted_index`, thus facilitating quick lookup and application of these results during execution.

airborne12 · 2024-09-27T03:30:17Z

run buildall

doris-robot · 2024-09-27T03:30:20Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

github-actions

clang-tidy made some suggestions

github-actions · 2024-09-27T03:38:56Z

be/src/vec/exprs/vcompound_pred.h

@@ -53,7 +54,107 @@ class VCompoundPred : public VectorizedFnCall {

    const std::string& expr_name() const override { return _expr_name; }

+    Status evaluate_inverted_index(VExprContext* context, uint32_t segment_num_rows) override {


warning: function 'evaluate_inverted_index' exceeds recommended size/complexity thresholds [readability-function-size]

Status evaluate_inverted_index(VExprContext* context, uint32_t segment_num_rows) override { ^

Additional context

be/src/vec/exprs/vcompound_pred.h:56: 95 lines including whitespace and comments (threshold 80)

Status evaluate_inverted_index(VExprContext* context, uint32_t segment_num_rows) override { ^

github-actions · 2024-09-27T03:38:56Z

be/src/vec/exprs/vcompound_pred.h

+        }
+        return Status::OK();
+    }
+
    Status execute(VExprContext* context, Block* block, int* result_column_id) override {


warning: function 'execute' has cognitive complexity of 112 (threshold 50) [readability-function-cognitive-complexity]

Status execute(VExprContext* context, Block* block, int* result_column_id) override { ^

Additional context

be/src/vec/exprs/vcompound_pred.h:154: +1, including nesting penalty of 0, nesting level increased to 1

if (_can_fast_execute && fast_execute(context, block, result_column_id)) { ^

be/src/vec/exprs/vcompound_pred.h:154: +1

if (_can_fast_execute && fast_execute(context, block, result_column_id)) { ^

be/src/vec/exprs/vcompound_pred.h:157: +1, including nesting penalty of 0, nesting level increased to 1

if (children().size() == 1 || !_all_child_is_compound_and_not_const()) { ^

be/src/vec/exprs/vcompound_pred.h:163: +1, including nesting penalty of 0, nesting level increased to 1

RETURN_IF_ERROR(_children[0]->execute(context, block, &lhs_id)); ^

be/src/common/status.h:619: expanded from macro 'RETURN_IF_ERROR'

do { \ ^

be/src/vec/exprs/vcompound_pred.h:163: +2, including nesting penalty of 1, nesting level increased to 2

RETURN_IF_ERROR(_children[0]->execute(context, block, &lhs_id)); ^

be/src/common/status.h:621: expanded from macro 'RETURN_IF_ERROR'

if (UNLIKELY(!_status_.ok())) { \ ^

be/src/vec/exprs/vcompound_pred.h:175: +1, including nesting penalty of 0, nesting level increased to 1

if (lhs_is_nullable) { ^

be/src/vec/exprs/vcompound_pred.h:189: nesting level increased to 1

auto get_rhs_colum = [&]() { ^

be/src/vec/exprs/vcompound_pred.h:190: +2, including nesting penalty of 1, nesting level increased to 2

if (rhs_id == -1) { ^

be/src/vec/exprs/vcompound_pred.h:191: +3, including nesting penalty of 2, nesting level increased to 3

RETURN_IF_ERROR(_children[1]->execute(context, block, &rhs_id)); ^

be/src/common/status.h:619: expanded from macro 'RETURN_IF_ERROR'

do { \ ^

be/src/vec/exprs/vcompound_pred.h:191: +4, including nesting penalty of 3, nesting level increased to 4

RETURN_IF_ERROR(_children[1]->execute(context, block, &rhs_id)); ^

be/src/common/status.h:621: expanded from macro 'RETURN_IF_ERROR'

if (UNLIKELY(!_status_.ok())) { \ ^

be/src/vec/exprs/vcompound_pred.h:201: +3, including nesting penalty of 2, nesting level increased to 3

if (rhs_is_nullable) { ^

be/src/vec/exprs/vcompound_pred.h:209: nesting level increased to 1

auto return_result_column_id = [&](ColumnPtr res_column, int res_id) -> int { ^

be/src/vec/exprs/vcompound_pred.h:210: +2, including nesting penalty of 1, nesting level increased to 2

if (result_is_nullable && !res_column->is_nullable()) { ^

be/src/vec/exprs/vcompound_pred.h:210: +1

if (result_is_nullable && !res_column->is_nullable()) { ^

be/src/vec/exprs/vcompound_pred.h:219: nesting level increased to 1

auto create_null_map_column = [&](ColumnPtr& null_map_column, ^

be/src/vec/exprs/vcompound_pred.h:221: +2, including nesting penalty of 1, nesting level increased to 2

if (null_map_data == nullptr) { ^

be/src/vec/exprs/vcompound_pred.h:230: nesting level increased to 1

auto vector_vector_null = [&]<bool is_and_op>() { ^

be/src/vec/exprs/vcompound_pred.h:240: +2, including nesting penalty of 1, nesting level increased to 2

if constexpr (is_and_op) { ^

be/src/vec/exprs/vcompound_pred.h:241: +3, including nesting penalty of 2, nesting level increased to 3

for (size_t i = 0; i < size; ++i) { ^

be/src/vec/exprs/vcompound_pred.h:246: +1, nesting level increased to 2

} else { ^

be/src/vec/exprs/vcompound_pred.h:247: +3, including nesting penalty of 2, nesting level increased to 3

for (size_t i = 0; i < size; ++i) { ^

be/src/vec/exprs/vcompound_pred.h:260: +1, including nesting penalty of 0, nesting level increased to 1

if (_op == TExprOpcode::COMPOUND_AND) { ^

be/src/vec/exprs/vcompound_pred.h:263: +2, including nesting penalty of 1, nesting level increased to 2

if ((lhs_all_false && !lhs_is_nullable) || (lhs_all_false && lhs_all_is_not_null)) { ^

be/src/vec/exprs/vcompound_pred.h:263: +1

if ((lhs_all_false && !lhs_is_nullable) || (lhs_all_false && lhs_all_is_not_null)) { ^

be/src/vec/exprs/vcompound_pred.h:263: +1

if ((lhs_all_false && !lhs_is_nullable) || (lhs_all_false && lhs_all_is_not_null)) { ^

be/src/vec/exprs/vcompound_pred.h:263: +1

if ((lhs_all_false && !lhs_is_nullable) || (lhs_all_false && lhs_all_is_not_null)) { ^

be/src/vec/exprs/vcompound_pred.h:266: +1, nesting level increased to 2

} else { ^

be/src/vec/exprs/vcompound_pred.h:267: +3, including nesting penalty of 2, nesting level increased to 3

RETURN_IF_ERROR(get_rhs_colum()); ^

be/src/common/status.h:619: expanded from macro 'RETURN_IF_ERROR'

do { \ ^

be/src/vec/exprs/vcompound_pred.h:267: +4, including nesting penalty of 3, nesting level increased to 4

RETURN_IF_ERROR(get_rhs_colum()); ^

be/src/common/status.h:621: expanded from macro 'RETURN_IF_ERROR'

if (UNLIKELY(!_status_.ok())) { \ ^

be/src/vec/exprs/vcompound_pred.h:269: +3, including nesting penalty of 2, nesting level increased to 3

if ((lhs_all_true && !lhs_is_nullable) || //not null column ^

be/src/vec/exprs/vcompound_pred.h:269: +1

if ((lhs_all_true && !lhs_is_nullable) || //not null column ^

be/src/vec/exprs/vcompound_pred.h:269: +1

if ((lhs_all_true && !lhs_is_nullable) || //not null column ^

be/src/vec/exprs/vcompound_pred.h:270: +1

(lhs_all_true && lhs_all_is_not_null)) { //nullable column ^

be/src/vec/exprs/vcompound_pred.h:273: +1, nesting level increased to 3

} else if ((rhs_all_false && !rhs_is_nullable) || ^

be/src/vec/exprs/vcompound_pred.h:273: +1

} else if ((rhs_all_false && !rhs_is_nullable) || ^

be/src/vec/exprs/vcompound_pred.h:273: +1

} else if ((rhs_all_false && !rhs_is_nullable) || ^

be/src/vec/exprs/vcompound_pred.h:274: +1

(rhs_all_false && rhs_all_is_not_null)) { ^

be/src/vec/exprs/vcompound_pred.h:277: +1, nesting level increased to 3

} else if ((rhs_all_true && !rhs_is_nullable) || ^

be/src/vec/exprs/vcompound_pred.h:277: +1

} else if ((rhs_all_true && !rhs_is_nullable) || ^

be/src/vec/exprs/vcompound_pred.h:277: +1

} else if ((rhs_all_true && !rhs_is_nullable) || ^

be/src/vec/exprs/vcompound_pred.h:278: +1

(rhs_all_true && rhs_all_is_not_null)) { ^

be/src/vec/exprs/vcompound_pred.h:281: +1, nesting level increased to 3

} else { ^

be/src/vec/exprs/vcompound_pred.h:282: +4, including nesting penalty of 3, nesting level increased to 4

if (!result_is_nullable) { ^

be/src/vec/exprs/vcompound_pred.h:284: +5, including nesting penalty of 4, nesting level increased to 5

for (size_t i = 0; i < size; i++) { ^

be/src/vec/exprs/vcompound_pred.h:287: +1, nesting level increased to 4

} else { ^

be/src/vec/exprs/vcompound_pred.h:292: +1, nesting level increased to 1

} else if (_op == TExprOpcode::COMPOUND_OR) { ^

be/src/vec/exprs/vcompound_pred.h:295: +2, including nesting penalty of 1, nesting level increased to 2

if ((lhs_all_true && !lhs_is_nullable) || (lhs_all_true && lhs_all_is_not_null)) { ^

be/src/vec/exprs/vcompound_pred.h:295: +1

if ((lhs_all_true && !lhs_is_nullable) || (lhs_all_true && lhs_all_is_not_null)) { ^

be/src/vec/exprs/vcompound_pred.h:295: +1

if ((lhs_all_true && !lhs_is_nullable) || (lhs_all_true && lhs_all_is_not_null)) { ^

be/src/vec/exprs/vcompound_pred.h:295: +1

if ((lhs_all_true && !lhs_is_nullable) || (lhs_all_true && lhs_all_is_not_null)) { ^

be/src/vec/exprs/vcompound_pred.h:298: +1, nesting level increased to 2

} else { ^

be/src/vec/exprs/vcompound_pred.h:299: +3, including nesting penalty of 2, nesting level increased to 3

RETURN_IF_ERROR(get_rhs_colum()); ^

be/src/common/status.h:619: expanded from macro 'RETURN_IF_ERROR'

do { \ ^

be/src/vec/exprs/vcompound_pred.h:299: +4, including nesting penalty of 3, nesting level increased to 4

RETURN_IF_ERROR(get_rhs_colum()); ^

be/src/common/status.h:621: expanded from macro 'RETURN_IF_ERROR'

if (UNLIKELY(!_status_.ok())) { \ ^

be/src/vec/exprs/vcompound_pred.h:300: +3, including nesting penalty of 2, nesting level increased to 3

if ((lhs_all_false && !lhs_is_nullable) || (lhs_all_false && lhs_all_is_not_null)) { ^

be/src/vec/exprs/vcompound_pred.h:300: +1

if ((lhs_all_false && !lhs_is_nullable) || (lhs_all_false && lhs_all_is_not_null)) { ^

be/src/vec/exprs/vcompound_pred.h:300: +1

if ((lhs_all_false && !lhs_is_nullable) || (lhs_all_false && lhs_all_is_not_null)) { ^

be/src/vec/exprs/vcompound_pred.h:300: +1

if ((lhs_all_false && !lhs_is_nullable) || (lhs_all_false && lhs_all_is_not_null)) { ^

be/src/vec/exprs/vcompound_pred.h:303: +1, nesting level increased to 3

} else if ((rhs_all_true && !rhs_is_nullable) || ^

be/src/vec/exprs/vcompound_pred.h:303: +1

} else if ((rhs_all_true && !rhs_is_nullable) || ^

be/src/vec/exprs/vcompound_pred.h:303: +1

} else if ((rhs_all_true && !rhs_is_nullable) || ^

be/src/vec/exprs/vcompound_pred.h:304: +1

(rhs_all_true && rhs_all_is_not_null)) { ^

be/src/vec/exprs/vcompound_pred.h:307: +1, nesting level increased to 3

} else if ((rhs_all_false && !rhs_is_nullable) || ^

be/src/vec/exprs/vcompound_pred.h:307: +1

} else if ((rhs_all_false && !rhs_is_nullable) || ^

be/src/vec/exprs/vcompound_pred.h:307: +1

} else if ((rhs_all_false && !rhs_is_nullable) || ^

be/src/vec/exprs/vcompound_pred.h:308: +1

(rhs_all_false && rhs_all_is_not_null)) { ^

be/src/vec/exprs/vcompound_pred.h:311: +1, nesting level increased to 3

} else { ^

be/src/vec/exprs/vcompound_pred.h:312: +4, including nesting penalty of 3, nesting level increased to 4

if (!result_is_nullable) { ^

be/src/vec/exprs/vcompound_pred.h:314: +5, including nesting penalty of 4, nesting level increased to 5

for (size_t i = 0; i < size; i++) { ^

be/src/vec/exprs/vcompound_pred.h:317: +1, nesting level increased to 4

} else { ^

be/src/vec/exprs/vcompound_pred.h:322: +1, nesting level increased to 1

} else { ^

github-actions · 2024-09-27T03:38:56Z

be/src/vec/exprs/vexpr.cpp

-                         size_t input_rows_count, const std::string& function_name) {
-    if (!_enable_inverted_index_query) {
-        return false;
+Status VExpr::_evaluate_inverted_index(VExprContext* context, const FunctionBasePtr& function,


warning: function '_evaluate_inverted_index' exceeds recommended size/complexity thresholds [readability-function-size]

Status VExpr::_evaluate_inverted_index(VExprContext* context, const FunctionBasePtr& function, ^

Additional context

be/src/vec/exprs/vexpr.cpp:604: 106 lines including whitespace and comments (threshold 80)

Status VExpr::_evaluate_inverted_index(VExprContext* context, const FunctionBasePtr& function, ^

github-actions · 2024-09-27T03:38:56Z

be/src/vec/functions/match.cpp

-        }
+    VLOG_DEBUG << "begin to execute match directly, column_name=" << column_name
+               << ", match_query_str=" << match_query_str;
+    InvertedIndexCtx* inverted_index_ctx = reinterpret_cast<InvertedIndexCtx*>(


warning: use auto when initializing with a cast to avoid duplicating the type name [modernize-use-auto]

Suggested change

InvertedIndexCtx* inverted_index_ctx = reinterpret_cast<InvertedIndexCtx*>(

auto* inverted_index_ctx = reinterpret_cast<InvertedIndexCtx*>(

airborne12 · 2024-09-27T07:27:24Z

run buildall

airborne12 · 2024-09-27T08:01:16Z

run buildall

airborne12 · 2024-09-27T08:34:07Z

run buildall

airborne12 · 2024-09-27T08:53:34Z

run buildall

airborne12 · 2024-09-27T09:16:28Z

run buildall

airborne12 · 2024-09-27T14:21:54Z

run buildall

doris-robot · 2024-09-27T15:32:35Z

TeamCity be ut coverage result:
Function Coverage: 36.06% (9344/25911)
Line Coverage: 27.62% (76752/277920)
Region Coverage: 26.39% (39380/149229)
Branch Coverage: 23.18% (20035/86450)
Coverage Report: http://coverage.selectdb-in.cc/coverage/9d1ad41f1de184322afb2e84d9dbd9da6e1b5cb2_9d1ad41f1de184322afb2e84d9dbd9da6e1b5cb2/report/index.html

airborne12 · 2024-09-28T03:38:50Z

run buildall

doris-robot · 2024-09-28T04:15:10Z

TeamCity be ut coverage result:
Function Coverage: 36.05% (9342/25911)
Line Coverage: 27.61% (76731/277918)
Region Coverage: 26.38% (39372/149229)
Branch Coverage: 23.17% (20034/86450)
Coverage Report: http://coverage.selectdb-in.cc/coverage/2edb5ff58c81aea8ac672a44acb6d618e2f4b51e_2edb5ff58c81aea8ac672a44acb6d618e2f4b51e/report/index.html

airborne12 · 2024-09-28T10:22:03Z

run buildall

doris-robot · 2024-09-28T10:58:59Z

TeamCity be ut coverage result:
Function Coverage: 36.06% (9344/25911)
Line Coverage: 27.62% (76762/277918)
Region Coverage: 26.39% (39385/149229)
Branch Coverage: 23.19% (20044/86450)
Coverage Report: http://coverage.selectdb-in.cc/coverage/3d020715f04067893cefa209631b48a13341168a_3d020715f04067893cefa209631b48a13341168a/report/index.html

github-actions bot reviewed Sep 27, 2024

View reviewed changes

Update vexpr_context.h

cfd0e00

Update segment_iterator.cpp

28f4ed5

Update in.h

4a87900

Update inverted_index_reader.h

c1068d8

Update inverted_index_reader.h

0bbfc41

fix compile

9d1ad41

fix coredump

2edb5ff

fix

3d02071

airborne12 merged commit 727f037 into apache:branch-2.1 Sep 29, 2024
19 of 21 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Refactor](inverted index) refactor inverted index compound predicates evaluate logic #38908 #41385

[Refactor](inverted index) refactor inverted index compound predicates evaluate logic #38908 #41385

airborne12 commented Sep 27, 2024

airborne12 commented Sep 27, 2024

doris-robot commented Sep 27, 2024

github-actions bot left a comment

github-actions bot Sep 27, 2024

github-actions bot Sep 27, 2024

github-actions bot Sep 27, 2024

github-actions bot Sep 27, 2024

airborne12 commented Sep 27, 2024

airborne12 commented Sep 27, 2024

airborne12 commented Sep 27, 2024

airborne12 commented Sep 27, 2024

airborne12 commented Sep 27, 2024

airborne12 commented Sep 27, 2024

doris-robot commented Sep 27, 2024

airborne12 commented Sep 28, 2024

doris-robot commented Sep 28, 2024

airborne12 commented Sep 28, 2024

doris-robot commented Sep 28, 2024

		@@ -53,7 +54,107 @@ class VCompoundPred : public VectorizedFnCall {

		const std::string& expr_name() const override { return _expr_name; }

		Status evaluate_inverted_index(VExprContext* context, uint32_t segment_num_rows) override {

	InvertedIndexCtx* inverted_index_ctx = reinterpret_cast<InvertedIndexCtx*>(
	auto* inverted_index_ctx = reinterpret_cast<InvertedIndexCtx*>(

[Refactor](inverted index) refactor inverted index compound predicates evaluate logic #38908 #41385

[Refactor](inverted index) refactor inverted index compound predicates evaluate logic #38908 #41385

Conversation

airborne12 commented Sep 27, 2024

airborne12 commented Sep 27, 2024

doris-robot commented Sep 27, 2024

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot Sep 27, 2024

Choose a reason for hiding this comment

github-actions bot Sep 27, 2024

Choose a reason for hiding this comment

github-actions bot Sep 27, 2024

Choose a reason for hiding this comment

github-actions bot Sep 27, 2024

Choose a reason for hiding this comment

airborne12 commented Sep 27, 2024

airborne12 commented Sep 27, 2024

airborne12 commented Sep 27, 2024

airborne12 commented Sep 27, 2024

airborne12 commented Sep 27, 2024

airborne12 commented Sep 27, 2024

doris-robot commented Sep 27, 2024

airborne12 commented Sep 28, 2024

doris-robot commented Sep 28, 2024

airborne12 commented Sep 28, 2024

doris-robot commented Sep 28, 2024