[opt](inverted index) add performance profiling for remote io access in inverted index #43542

zzzxl1993 · 2024-11-10T13:04:41Z

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Implement the accounting of remote I/O operations for the inverted index in profiling statistics.

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

doris-robot · 2024-11-10T13:04:46Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

github-actions

clang-tidy made some suggestions

github-actions · 2024-11-10T13:10:00Z

be/src/olap/rowset/segment_v2/inverted_index_reader.cpp

-Status FullTextIndexReader::query(OlapReaderStatistics* stats, RuntimeState* runtime_state,
-                                  const std::string& column_name, const void* query_value,
-                                  InvertedIndexQueryType query_type,
+Status FullTextIndexReader::query(const io::IOContext* io_ctx, OlapReaderStatistics* stats,


warning: function 'query' exceeds recommended size/complexity thresholds [readability-function-size]

Status FullTextIndexReader::query(const io::IOContext* io_ctx, OlapReaderStatistics* stats, ^

Additional context

be/src/olap/rowset/segment_v2/inverted_index_reader.cpp:250: 81 lines including whitespace and comments (threshold 80)

Status FullTextIndexReader::query(const io::IOContext* io_ctx, OlapReaderStatistics* stats, ^

github-actions · 2024-11-10T13:10:00Z

be/src/olap/rowset/segment_v2/inverted_index_reader.cpp

 }

-Status StringTypeInvertedIndexReader::query(OlapReaderStatistics* stats,
+Status StringTypeInvertedIndexReader::query(const io::IOContext* io_ctx,


warning: function 'query' exceeds recommended size/complexity thresholds [readability-function-size]

Status StringTypeInvertedIndexReader::query(const io::IOContext* io_ctx, ^

Additional context

be/src/olap/rowset/segment_v2/inverted_index_reader.cpp:348: 117 lines including whitespace and comments (threshold 80)

Status StringTypeInvertedIndexReader::query(const io::IOContext* io_ctx, ^

zzzxl1993 · 2024-11-11T07:02:12Z

run buildall

zzzxl1993 · 2024-11-11T07:14:09Z

run buildall

be/src/olap/rowset/segment_v2/inverted_index_file_reader.h

zzzxl1993 · 2024-11-12T11:15:05Z

run buildall

zzzxl1993 · 2024-11-13T02:34:09Z

run buildall

zzzxl1993 · 2024-11-13T03:16:49Z

run buildall

zzzxl1993 · 2024-11-13T05:32:36Z

run buildall

zzzxl1993 · 2024-11-13T07:48:47Z

run buildall

airborne12

LGTM

github-actions · 2024-11-13T08:47:59Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-11-13T08:48:01Z

PR approved by anyone and no changes requested.

zzzxl1993 · 2024-11-13T11:20:55Z

run buildall

zzzxl1993 · 2024-11-14T09:28:37Z

run buildall

github-actions

clang-tidy made some suggestions

github-actions · 2024-11-14T09:33:29Z

be/src/olap/rowset/segment_v2/inverted_index_file_reader.cpp

 }

-Status InvertedIndexFileReader::_init_from_v2(int32_t read_buffer_size) {
+Status InvertedIndexFileReader::_init_from_v2(const io::IOContext* io_ctx,


warning: function '_init_from_v2' exceeds recommended size/complexity thresholds [readability-function-size]

Status InvertedIndexFileReader::_init_from_v2(const io::IOContext* io_ctx, ^

Additional context

be/src/olap/rowset/segment_v2/inverted_index_file_reader.cpp:43: 94 lines including whitespace and comments (threshold 80)

Status InvertedIndexFileReader::_init_from_v2(const io::IOContext* io_ctx, ^

doris-robot · 2024-11-14T11:13:58Z

TeamCity be ut coverage result:
Function Coverage: 37.95% (9891/26065)
Line Coverage: 29.14% (82650/283622)
Region Coverage: 28.27% (42481/150261)
Branch Coverage: 24.85% (21547/86692)
Coverage Report: http://coverage.selectdb-in.cc/coverage/521ef057683eb938723c89068dd5896e0eab752f_521ef057683eb938723c89068dd5896e0eab752f/report/index.html

zzzxl1993 · 2024-11-14T15:40:48Z

run buildall

zzzxl1993 · 2024-11-14T15:40:59Z

run buildall

zzzxl1993 · 2024-11-15T04:25:37Z

run p0

github-actions

clang-tidy made some suggestions

github-actions · 2024-11-15T04:31:16Z

be/src/olap/rowset/segment_v2/inverted_index_reader.cpp

-Status FullTextIndexReader::query(OlapReaderStatistics* stats, RuntimeState* runtime_state,
-                                  const std::string& column_name, const void* query_value,
-                                  InvertedIndexQueryType query_type,
+Status FullTextIndexReader::query(const io::IOContext* io_ctx, OlapReaderStatistics* stats,


warning: function 'query' exceeds recommended size/complexity thresholds [readability-function-size]

Status FullTextIndexReader::query(const io::IOContext* io_ctx, OlapReaderStatistics* stats, ^

Additional context

be/src/olap/rowset/segment_v2/inverted_index_reader.cpp:248: 81 lines including whitespace and comments (threshold 80)

Status FullTextIndexReader::query(const io::IOContext* io_ctx, OlapReaderStatistics* stats, ^

github-actions · 2024-11-15T04:31:16Z

be/src/olap/rowset/segment_v2/inverted_index_reader.cpp

 }

-Status StringTypeInvertedIndexReader::query(OlapReaderStatistics* stats,
+Status StringTypeInvertedIndexReader::query(const io::IOContext* io_ctx,


warning: function 'query' exceeds recommended size/complexity thresholds [readability-function-size]

Status StringTypeInvertedIndexReader::query(const io::IOContext* io_ctx, ^

Additional context

be/src/olap/rowset/segment_v2/inverted_index_reader.cpp:346: 117 lines including whitespace and comments (threshold 80)

Status StringTypeInvertedIndexReader::query(const io::IOContext* io_ctx, ^

zzzxl1993 · 2024-11-17T03:01:11Z

run buildall

…in inverted index

zzzxl1993 · 2024-11-17T03:17:23Z

run buildall

doris-robot · 2024-11-17T04:55:39Z

TeamCity be ut coverage result:
Function Coverage: 37.97% (9899/26074)
Line Coverage: 29.17% (82747/283683)
Region Coverage: 28.30% (42486/150132)
Branch Coverage: 24.87% (21542/86606)
Coverage Report: http://coverage.selectdb-in.cc/coverage/636764ebfe6c4361fb5e19ba4bb7fb0fa90fd2cd_636764ebfe6c4361fb5e19ba4bb7fb0fa90fd2cd/report/index.html

csun5285

LGTM

qidaye

LGTM

github-actions · 2024-11-17T13:44:33Z

PR approved by at least one committer and no changes requested.

…in inverted index (#43542) Problem Summary: 1. Implement the accounting of remote I/O operations for the inverted index in profiling statistics. ### Release note Add performance profiling for remote IO access in inverted index

…e io access in inverted index #43542 (#44093) Cherry-picked from #43542 Co-authored-by: zzzxl <yangsiyu@selectdb.com>

… profile (#44863) Add NumInvertedIndexRemoteIOTotal to count the number of remote IO operations in the inverted index Related PR: #43542

… profile (apache#44863) Add NumInvertedIndexRemoteIOTotal to count the number of remote IO operations in the inverted index Related PR: apache#43542

github-actions bot reviewed Nov 10, 2024

View reviewed changes

zzzxl1993 force-pushed the 202411052006 branch 2 times, most recently from 028c8fd to 6e99cbb Compare November 11, 2024 07:00

csun5285 reviewed Nov 12, 2024

View reviewed changes

be/src/olap/rowset/segment_v2/inverted_index_file_reader.h Outdated Show resolved Hide resolved

zzzxl1993 force-pushed the 202411052006 branch 2 times, most recently from 025f6e7 to c0b7127 Compare November 12, 2024 11:14

zzzxl1993 force-pushed the 202411052006 branch from c0b7127 to d034bf8 Compare November 13, 2024 02:33

zzzxl1993 force-pushed the 202411052006 branch from d034bf8 to fde1fb1 Compare November 13, 2024 03:16

zzzxl1993 force-pushed the 202411052006 branch from fde1fb1 to 7467f40 Compare November 13, 2024 05:32

zzzxl1993 force-pushed the 202411052006 branch from 7467f40 to dc4ab34 Compare November 13, 2024 07:48

airborne12 previously approved these changes Nov 13, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Nov 13, 2024

github-actions bot added the reviewed label Nov 13, 2024

zzzxl1993 dismissed airborne12’s stale review via 8a9bd64 November 13, 2024 11:12

zzzxl1993 force-pushed the 202411052006 branch from dc4ab34 to 8a9bd64 Compare November 13, 2024 11:12

github-actions bot removed the approved Indicates a PR has been approved by one committer. label Nov 13, 2024

zzzxl1993 force-pushed the 202411052006 branch from 8a9bd64 to 521ef05 Compare November 14, 2024 09:27

github-actions bot reviewed Nov 14, 2024

View reviewed changes

zzzxl1993 force-pushed the 202411052006 branch from 521ef05 to d7bab6b Compare November 14, 2024 15:33

zzzxl1993 closed this Nov 14, 2024

zzzxl1993 reopened this Nov 14, 2024

zzzxl1993 force-pushed the 202411052006 branch from d7bab6b to 7b786c9 Compare November 15, 2024 04:25

github-actions bot reviewed Nov 15, 2024

View reviewed changes

zzzxl1993 force-pushed the 202411052006 branch from 7b786c9 to 4d85d7e Compare November 17, 2024 03:00

[opt](inverted index) add performance profiling for remote io access …

636764e

…in inverted index

zzzxl1993 force-pushed the 202411052006 branch from 4d85d7e to 636764e Compare November 17, 2024 03:17

csun5285 approved these changes Nov 17, 2024

View reviewed changes

qidaye approved these changes Nov 17, 2024

View reviewed changes

qidaye added the dev/3.0.x label Nov 17, 2024

github-actions bot added the approved Indicates a PR has been approved by one committer. label Nov 17, 2024

qidaye merged commit 74238ae into apache:master Nov 17, 2024
27 of 29 checks passed

github-actions bot mentioned this pull request Nov 17, 2024

branch-3.0: [opt](inverted index) add performance profiling for remote io access in inverted index #43542 #44093

Merged

airborne12 pushed a commit that referenced this pull request Nov 18, 2024

branch-3.0: [opt](inverted index) add performance profiling for remot…

fdbd9ca

…e io access in inverted index #43542 (#44093) Cherry-picked from #43542 Co-authored-by: zzzxl <yangsiyu@selectdb.com>

airborne12 added dev/3.0.3-merged and removed dev/3.0.x labels Nov 18, 2024

zzzxl1993 mentioned this pull request Dec 3, 2024

[opt](inverted index) Add NumInvertedIndexRemoteIOTotal statistics in profile #44863

Merged

16 tasks

gavinchou pushed a commit that referenced this pull request Dec 13, 2024

[opt](inverted index) Add NumInvertedIndexRemoteIOTotal statistics in…

559a2f9

… profile (#44863) Add NumInvertedIndexRemoteIOTotal to count the number of remote IO operations in the inverted index Related PR: #43542

[opt](inverted index) add performance profiling for remote io access in inverted index #43542

[opt](inverted index) add performance profiling for remote io access in inverted index #43542

Uh oh!

Conversation

zzzxl1993 commented Nov 10, 2024 • edited by qidaye Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

doris-robot commented Nov 10, 2024

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot Nov 10, 2024

Choose a reason for hiding this comment

Uh oh!

github-actions bot Nov 10, 2024

Choose a reason for hiding this comment

Uh oh!

zzzxl1993 commented Nov 11, 2024

Uh oh!

zzzxl1993 commented Nov 11, 2024

Uh oh!

Uh oh!

zzzxl1993 commented Nov 12, 2024

Uh oh!

zzzxl1993 commented Nov 13, 2024

Uh oh!

zzzxl1993 commented Nov 13, 2024

Uh oh!

zzzxl1993 commented Nov 13, 2024

Uh oh!

zzzxl1993 commented Nov 13, 2024

Uh oh!

airborne12 left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Nov 13, 2024

Uh oh!

github-actions bot commented Nov 13, 2024

Uh oh!

zzzxl1993 commented Nov 13, 2024

Uh oh!

zzzxl1993 commented Nov 14, 2024

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot Nov 14, 2024

Choose a reason for hiding this comment

Uh oh!

doris-robot commented Nov 14, 2024

Uh oh!

zzzxl1993 commented Nov 14, 2024

Uh oh!

zzzxl1993 commented Nov 14, 2024

Uh oh!

zzzxl1993 commented Nov 15, 2024

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot Nov 15, 2024

Choose a reason for hiding this comment

Uh oh!

github-actions bot Nov 15, 2024

Choose a reason for hiding this comment

Uh oh!

zzzxl1993 commented Nov 17, 2024

Uh oh!

zzzxl1993 commented Nov 17, 2024

Uh oh!

doris-robot commented Nov 17, 2024

Uh oh!

csun5285 left a comment

Choose a reason for hiding this comment

Uh oh!

zzzxl1993 commented Nov 10, 2024 •

edited by qidaye

Loading