Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

executor: fix distorted join probe time #52227

Merged
merged 5 commits into from Apr 7, 2024

Conversation

windtalker
Copy link
Contributor

@windtalker windtalker commented Mar 29, 2024

What problem does this PR solve?

Issue Number: close #52222

Problem Summary:

What changed and how does it work?

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
    run TPCH query 10 base on TPCH-50 dataset
    Before this pr
|           ├─HashJoin_88(Build)                | 2882295.78  | 2865763  | root      |                                                         | time:4.93s, loops:2802, build_hash_table:{total:160.6µs, fetch:135.6µs, build:25µs}, probe:{concurrency:5, total:50.9s, max:10.2s, probe:27.9s, fetch:23s}                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             | inner join, equal:[eq(test.customer.c_nationkey, test.nation.n_nationkey)]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            | 28.5 KB   | 0 Bytes |
|           │ └─HashJoin_99(Probe)              | 2882295.78  | 2865763  | root      |                                                         | time:5.18s, loops:2801, build_hash_table:{total:3.72s, fetch:2.59s, build:1.12s}, probe:{concurrency:5, total:50.9s, max:10.2s, probe:32.2s, fetch:18.7s}                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              | inner join, equal:[eq(test.customer.c_custkey, test.orders.o_custkey)]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                | 273.0 MB  | 0 Bytes |

The displayed probe time 27.9s for HashJoin_88 is 27.9s and 32.2s for HashJoin_99
After this pr

|           ├─HashJoin_88(Build)                | 2882295.78  | 2865763  | root      |                                                         | time:4.78s, loops:2802, build_hash_table:{total:292.5µs, fetch:277.3µs, build:15.2µs}, probe:{concurrency:5, total:48.6s, max:9.71s, probe:3.16s, fetch and wait:45.4s}                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  | inner join, equal:[eq(test.customer.c_nationkey, test.nation.n_nationkey)]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            | 35.2 KB   | 0 Bytes |                                                                                                                                                                                                                                                                                                                                                                                                                                              
|           │ └─HashJoin_99(Probe)              | 2882295.78  | 2865763  | root      |                                                         | time:5.01s, loops:2801, build_hash_table:{total:3.62s, fetch:2.52s, build:1.11s}, probe:{concurrency:5, total:48.6s, max:9.71s, probe:7.72s, fetch and wait:40.8s}                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       | inner join, equal:[eq(test.customer.c_custkey, test.orders.o_custkey)]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                | 273.2 MB  | 0 Bytes |

The displayed probe time is 3.16s for HashJoin_88 and 7.72s for HashJoin_99

  • No need to test
    • I checked and no code files have been changed.

Side effects

  • Performance regression: Consumes more CPU
  • Performance regression: Consumes more Memory
  • Breaking backward compatibility

Documentation

  • Affects user behaviors
  • Contains syntax changes
  • Contains variable changes
  • Contains experimental features
  • Changes MySQL compatibility

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

@ti-chi-bot ti-chi-bot bot added do-not-merge/invalid-title release-note-none size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Mar 29, 2024
Copy link

tiprow bot commented Mar 29, 2024

Hi @windtalker. Thanks for your PR.

PRs from untrusted users cannot be marked as trusted with /ok-to-test in this repo meaning untrusted PR authors can never trigger tests themselves. Collaborators can still trigger tests on the PR using /test all.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@windtalker windtalker changed the title fix distorted join probe time executor: fix distorted join probe time Mar 29, 2024
@windtalker windtalker force-pushed the fix_distorted_join_probe_time branch from 61c1ac4 to bd126f6 Compare March 29, 2024 07:01
Copy link

codecov bot commented Mar 29, 2024

Codecov Report

Merging #52227 (bfd2beb) into master (9ad0862) will increase coverage by 0.1767%.
Report is 3 commits behind head on master.
The diff coverage is 55.3030%.

Additional details and impacted files
@@               Coverage Diff                @@
##             master     #52227        +/-   ##
================================================
+ Coverage   74.1468%   74.3235%   +0.1767%     
================================================
  Files          1467       1467                
  Lines        354142     426737     +72595     
================================================
+ Hits         262585     317166     +54581     
- Misses        72388      89733     +17345     
- Partials      19169      19838       +669     
Flag Coverage Δ
integration 48.7819% <53.7878%> (?)
unit 70.9680% <49.2424%> (-1.9366%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components Coverage Δ
dumpling 53.9957% <ø> (-2.3014%) ⬇️
parser ∅ <ø> (∅)
br 49.3738% <ø> (+7.2227%) ⬆️

@windtalker windtalker force-pushed the fix_distorted_join_probe_time branch 2 times, most recently from 8caf208 to f876b82 Compare April 1, 2024 05:05
@@ -985,12 +994,14 @@ func (w *probeWorker) getNewJoinResult() (bool, *hashjoinWorkerResult) {
}

func (w *probeWorker) join2Chunk(probeSideChk *chunk.Chunk, hCtx *hashContext, joinResult *hashjoinWorkerResult,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add a comment for waitTime ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

@windtalker windtalker force-pushed the fix_distorted_join_probe_time branch from f876b82 to 8c5939f Compare April 1, 2024 13:01
@@ -1633,7 +1661,7 @@ func (e *hashJoinRuntimeStats) String() string {
buf.WriteString(execdetails.FormatDuration(time.Duration(atomic.LoadInt64(&e.maxFetchAndProbe))))
buf.WriteString(", probe:")
buf.WriteString(execdetails.FormatDuration(time.Duration(e.probe)))
buf.WriteString(", fetch:")
buf.WriteString(", fetch and wait:")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add a comment for what does fetch and wait mean?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

@windtalker windtalker force-pushed the fix_distorted_join_probe_time branch from 8c5939f to bfd2beb Compare April 3, 2024 04:59
@ti-chi-bot ti-chi-bot bot added approved needs-1-more-lgtm needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. labels Apr 3, 2024
windtalker and others added 5 commits April 7, 2024 09:49
Signed-off-by: xufei <xufeixw@mail.ustc.edu.cn>
Signed-off-by: xufei <xufeixw@mail.ustc.edu.cn>
Signed-off-by: xufei <xufeixw@mail.ustc.edu.cn>
Signed-off-by: xufei <xufei@pingcap.com>
Signed-off-by: xufei <xufeixw@mail.ustc.edu.cn>
@windtalker windtalker force-pushed the fix_distorted_join_probe_time branch from bfd2beb to 977ee2c Compare April 7, 2024 01:52
@ti-chi-bot ti-chi-bot bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Apr 7, 2024
Copy link

ti-chi-bot bot commented Apr 7, 2024

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: wshwsh12, XuHuaiyu

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Copy link

ti-chi-bot bot commented Apr 7, 2024

[LGTM Timeline notifier]

Timeline:

  • 2024-04-03 08:36:39.531306442 +0000 UTC m=+433061.058846988: ☑️ agreed by XuHuaiyu.
  • 2024-04-07 02:36:52.441082603 +0000 UTC m=+757073.968623147: ☑️ agreed by wshwsh12.

@ti-chi-bot ti-chi-bot bot merged commit d2eb902 into pingcap:master Apr 7, 2024
23 checks passed
@windtalker windtalker deleted the fix_distorted_join_probe_time branch April 8, 2024 03:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved lgtm release-note-none size/L Denotes a PR that changes 100-499 lines, ignoring generated files.
Projects
None yet
3 participants