Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Runtime Filter: Support Local Mode and In Type #7218

Merged
merged 5 commits into from May 29, 2023
Merged

Runtime Filter: Support Local Mode and In Type #7218

merged 5 commits into from May 29, 2023

Conversation

elsa0520
Copy link
Contributor

@elsa0520 elsa0520 commented Apr 4, 2023 •
edited

What problem does this PR solve?

Issue Number: close #7275

Problem Summary:

What is changed and how it works?

Complete the Runtime Filter Local mode, and implement the execution layer of the IN type.

Process：

  1. compile: Parse runtime filter for Join and UnorderedInputStream
  2. execute join input stream: Each new batch on the build side will update the value of the runtime filter until the last batch is read.
  3. execute unordered input stream: Generate runtime filter to IN RsOperator and push it to storage engine.

new variable

  1. "max_wait_time_ms" : Maximum waiting time (controlled by tidb)
  2. "RF_MAX_IN_VALUE_SET" : The maximum number of IN values: currently hard-coded 1024, the subsequent pr will be adjusted as a parameter.

Check List

Tests

!! UT WIP !!

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
  • No code

Side effects

  • Performance regression: Consumes more CPU
  • Performance regression: Consumes more Memory
  • Breaking backward compatibility

Documentation

  • Affects user behaviors
  • Contains syntax changes
  • Contains variable changes
  • Contains experimental features
  • Changes MySQL compatibility

Release note

None

@ti-chi-bot
Copy link
Member

ti-chi-bot commented Apr 4, 2023 •
edited by ti-chi-bot bot

[REVIEW NOTIFICATION]

This pull request has been approved by:

  • Lloyd-Pottiger
  • windtalker

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

@ti-chi-bot ti-chi-bot added do-not-merge/needs-linked-issue release-note-none do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Apr 4, 2023
@sre-bot
Copy link
Collaborator

sre-bot commented Apr 4, 2023 •
edited

CLA assistant check
All committers have signed the CLA.

@elsa0520 elsa0520 marked this pull request as ready for review April 4, 2023 12:12
@ti-chi-bot ti-chi-bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Apr 4, 2023
@elsa0520 elsa0520 changed the title Runtime Filter support In Runtime Filter: Support Local Mode and In Type Apr 4, 2023
@Lloyd-Pottiger Lloyd-Pottiger self-requested a review April 6, 2023 02:19
@ywqzzy ywqzzy self-requested a review April 6, 2023 08:48
flowbehappy
flowbehappy reviewed Apr 6, 2023
View reviewed changes
dbms/src/DataStreams/RuntimeFilter.cpp Outdated Show resolved Hide resolved
dbms/src/DataStreams/RuntimeFilter.cpp Outdated Show resolved Hide resolved
@Lloyd-Pottiger Lloyd-Pottiger removed their request for review April 7, 2023 01:59
ywqzzy
ywqzzy reviewed Apr 7, 2023
View reviewed changes
dbms/src/Flash/Coprocessor/DAGContext.h Outdated Show resolved Hide resolved
dbms/src/DataStreams/RuntimeFilter.cpp Outdated Show resolved Hide resolved
dbms/src/DataStreams/RuntimeFilter.cpp Outdated Show resolved Hide resolved
dbms/src/DataStreams/RuntimeFilter.cpp Outdated Show resolved Hide resolved
dbms/src/Interpreters/Join.cpp Outdated Show resolved Hide resolved
dbms/src/Flash/Coprocessor/DAGExpressionAnalyzer.cpp Show resolved Hide resolved
@ywqzzy ywqzzy self-requested a review April 7, 2023 07:04
ywqzzy
ywqzzy reviewed Apr 7, 2023
View reviewed changes
dbms/src/DataStreams/RuntimeFilter.h Outdated Show resolved Hide resolved
dbms/src/Flash/Coprocessor/RuntimeFilterMgr.h Show resolved Hide resolved
dbms/src/Flash/Coprocessor/JoinInterpreterHelper.cpp Outdated Show resolved Hide resolved
dbms/src/Flash/Coprocessor/RuntimeFilterMgr.h Outdated Show resolved Hide resolved
dbms/src/Flash/Coprocessor/RuntimeFilterMgr.h Outdated Show resolved Hide resolved
@ywqzzy ywqzzy self-requested a review April 7, 2023 07:24
ywqzzy
ywqzzy reviewed Apr 7, 2023
View reviewed changes
dbms/src/DataStreams/RuntimeFilter.cpp Outdated Show resolved Hide resolved
dbms/src/Storages/DeltaMerge/Index/MinMaxIndex.cpp Outdated Show resolved Hide resolved
dbms/src/DataStreams/RuntimeFilter.cpp Outdated Show resolved Hide resolved
dbms/src/DataStreams/RuntimeFilter.cpp Outdated Show resolved Hide resolved
@ti-chi-bot ti-chi-bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Apr 8, 2023
@ti-chi-bot ti-chi-bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Apr 10, 2023
@ti-chi-bot ti-chi-bot bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label May 26, 2023
@elsa0520
Copy link
Contributor Author

/run-integration-test

@elsa0520
Copy link
Contributor Author

/run-integration-test

1 similar comment
@elsa0520
Copy link
Contributor Author

/run-integration-test

@elsa0520
Copy link
Contributor Author

/run-unit-test

@elsa0520
Copy link
Contributor Author

/merge

@ti-chi-bot
Copy link
Contributor

ti-chi-bot bot commented May 26, 2023

@elsa0520: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

@ti-chi-bot
Copy link
Contributor

ti-chi-bot bot commented May 26, 2023

@elsa0520: /merge is only allowed for the committers, you can assign this pull request to the committer in list by filling /assign @committer in the comment to help merge this pull request.

In response to this:

/merge

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

@elsa0520
Copy link
Contributor Author

/run-all-tests

Lloyd-Pottiger
Lloyd-Pottiger reviewed May 26, 2023
View reviewed changes
dbms/src/Storages/DeltaMerge/File/DMFilePackFilter.h Outdated Show resolved Hide resolved
dbms/src/Storages/DeltaMerge/ReadThread/UnorderedInputStream.cpp Outdated Show resolved Hide resolved
dbms/src/Storages/DeltaMerge/tests/gtest_dm_storage_delta_merge.cpp
Comment on lines +788 to +794
query_info.dag_query = std::make_unique<DAGQueryInfo>(
google::protobuf::RepeatedPtrField<tipb::Expr>(),
pushed_down_filters, // Not care now
std::vector<TiDB::ColumnInfo>{}, // Not care now
std::vector<int>{},
0,
ctx->getTimezoneInfo());
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

seems can remove?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It should be keep otherwise the UT could not pass

@elsa0520
Copy link
Contributor Author

/run-all-tests

@Lloyd-Pottiger
Copy link
Contributor

/merge

@ti-chi-bot
Copy link
Contributor

ti-chi-bot bot commented May 29, 2023

@Lloyd-Pottiger: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

@ti-chi-bot
Copy link
Contributor

ti-chi-bot bot commented May 29, 2023

This pull request has been accepted and is ready to merge.

Commit hash: 5a1c9e7

@ti-chi-bot ti-chi-bot bot added the status/can-merge Indicates a PR has been approved by a committer. label May 29, 2023
@elsa0520
Copy link
Contributor Author

/merge

@ti-chi-bot
Copy link
Contributor

ti-chi-bot bot commented May 29, 2023

@elsa0520: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

@ti-chi-bot
Copy link
Contributor

ti-chi-bot bot commented May 29, 2023

@elsa0520: /merge is only allowed for the committers, you can assign this pull request to the committer in list by filling /assign @committer in the comment to help merge this pull request.

In response to this:

/merge

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

@elsa0520
Copy link
Contributor Author

/run-unit-test

@elsa0520
Copy link
Contributor Author

/assign @Lloyd-Pottiger

@ti-chi-bot ti-chi-bot bot merged commit 308d292 into pingcap:master May 29, 2023
6 checks passed
@SeaRise SeaRise mentioned this pull request Jun 26, 2023
Merged
12 tasks
@JaySon-Huang JaySon-Huang mentioned this pull request Oct 30, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@windtalker windtalker windtalker approved these changes

@flowbehappy flowbehappy flowbehappy left review comments

@yibin87 yibin87 yibin87 left review comments

@SeaRise SeaRise SeaRise left review comments

@Lloyd-Pottiger Lloyd-Pottiger Lloyd-Pottiger approved these changes

@ywqzzy ywqzzy Awaiting requested review from ywqzzy

Assignees

@Lloyd-Pottiger Lloyd-Pottiger

Labels
release-note-none size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. status/can-merge Indicates a PR has been approved by a committer. status/LGT2 Indicates that a PR has LGTM 2.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Runtime Filter: The executor layer of Runtime Filter
9 participants
@elsa0520 @ti-chi-bot @sre-bot @Lloyd-Pottiger @windtalker @flowbehappy @yibin87 @SeaRise @ywqzzy