explain: Add joins #3922

ghost · 2020-09-17T02:10:37Z

What is changed, added or deleted? (Required)

This is a followup PR to #3858 - but they can be reviewed independently, and merge out of order.

This PR does not add this page to the TOC. I will only do that once all PRs are merged (this is one ~3-4 followup PRs).

The content here is based on/duplicates query-execution-plan. I will merge all of the replacement pages, and then update query-execution-plan (soon to be called explain-overview) as a last step, and in the same PR which adds this page to the TOC.

Which TiDB version(s) do your changes apply to? (Required)

What is the related PR or file link(s)?

This PR is translated from:
Other reference link(s):

Do your changes match any of the following descriptions?

Delete files
Change aliases
Have version specific changes
Might cause conflicts

…in-joins

ghost · 2020-09-17T19:03:00Z

I have improved the examples to be reproducible. ~~I would like to expand this to explain more about hash joins, but I hit a bug on pingcap/tidb#20085~~ This has been fixed. The examples now show a hash join on disk.

explain-joins.md

ti-srebot · 2020-09-20T17:05:05Z

@SunRunAway, @zz-jason, @TomShawn, PTAL.

SunRunAway · 2020-09-21T09:21:46Z

Should the related variables be involved? like tidb_hash_join_concurrency, tidb_index_join_batch_size, tidb_index_lookup_join_concurrency

ghost · 2020-09-22T02:40:49Z

Should the related variables be involved? like tidb_hash_join_concurrency, tidb_index_join_batch_size, tidb_index_lookup_join_concurrency

@SunRunAway I have added this in ad7c61d PTAL. Thx :-)

ti-srebot · 2020-09-24T02:41:05Z

@SunRunAway, @zz-jason, @TomShawn, PTAL.

zz-jason · 2020-09-25T02:49:52Z

explain-joins.md

+
+### Runtime Statistics
+
+If [`tidb_mem_quota_query`](/system-variables.md#tidb_mem_quota_query) (default: 1GB) is exceeded, TiDB will attempt to use temporary storage provided that `oom-use-tmp-storage=TRUE` (default). This means that the hash table used as part of the hash join is created on disk. Runtime statistics, such as memory usage are visible in the `execution info` of `EXPLAIN ANALYZE`. The following example shows the output of `EXPLAIN ANALYZE` with a 1GB (default) `tidb_mem_quota_query` quota, and a 500MB quota. At 500MB the hash table spills to disk:


This means that the hash table used as part of the hash join is created on disk.

@SunRunAway please confirm: is the build side data cached in memory spilled to disk or the hash table itself is spilled to disk?

I think we should update https://docs.pingcap.com/tidb/stable/system-variables#tidb_mem_quota_query as well so its a bit clearer what this does.

For now, it is that build side data cached in memory spilled to disk, but we should also consider the spilling for hash table in the future.

It's better to clearly describe the current behavior on the document. we can update the document once we spilled hash table to disk.

BTW, for performance reason, it's not recommended to spill hash table to disk. There should be other methods to achieve the same goal.

I have attempted to clarify this in bcb16dd - PTAL, thx :-)

Co-authored-by: TomShawn <41534398+TomShawn@users.noreply.github.com>

TomShawn

LGTM

ti-srebot · 2020-11-04T03:53:30Z

/run-all-tests

ti-srebot · 2020-11-04T03:55:35Z

@nullnotnil merge failed.

Signed-off-by: ti-srebot <ti-srebot@pingcap.com>

ti-srebot · 2020-11-04T04:43:10Z

cherry pick to release-4.0 in PR #4147

TomShawn · 2020-11-04T04:53:23Z

Follow-up: TOC

Signed-off-by: ti-srebot <ti-srebot@pingcap.com> Co-authored-by: Null not nil <67764674+nullnotnil@users.noreply.github.com>

explain: Add joins

eccecee

ghost added status/PTAL This PR is ready for reviewing. needs-cherry-pick-4.0 labels Sep 17, 2020

ghost mentioned this pull request Sep 17, 2020

sql tuning: add EXPLAIN views #3862

Merged

9 tasks

TomShawn requested review from SunRunAway, zz-jason and TomShawn September 17, 2020 02:38

TomShawn self-assigned this Sep 17, 2020

TomShawn added size/large Changes of a large size. translation/doing This PR's assignee is translating this PR. labels Sep 17, 2020

Null not nil and others added 3 commits September 16, 2020 20:57

Merge branch 'master' into explain-joins

38cc519

Improve examples

282ed02

Merge branch 'explain-joins' of github.com:nullnotnil/docs into expla…

e8b4998

…in-joins

Update examples to show hash join on disk

932c399

SunRunAway reviewed Sep 18, 2020

View reviewed changes

explain-joins.md Outdated Show resolved Hide resolved

explain-joins.md Outdated Show resolved Hide resolved

explain-joins.md Outdated Show resolved Hide resolved

explain-joins.md Outdated Show resolved Hide resolved

Null not nil added 3 commits September 20, 2020 16:46

Merge remote-tracking branch 'upstream/master' into explain-joins

8691158

Add case to describe hash join vs index join

80ec5f2

Add suggestion on inner join types

1db44f1

TomShawn assigned ireneontheway and TomShawn and unassigned TomShawn and ireneontheway Sep 21, 2020

Add configuration

ad7c61d

ghost mentioned this pull request Sep 23, 2020

explain: add subqueries #3948

Merged

9 tasks

zz-jason reviewed Sep 25, 2020

View reviewed changes

Null not nil and others added 2 commits November 3, 2020 07:28

Update explain-joins.md

1b0d96e

Co-authored-by: TomShawn <41534398+TomShawn@users.noreply.github.com>

unify capitalization

d492704

TomShawn approved these changes Nov 4, 2020

View reviewed changes

ti-srebot removed the status/LGT1 Indicates that a PR has LGTM 1. label Nov 4, 2020

ti-srebot approved these changes Nov 4, 2020

View reviewed changes

ti-srebot added the status/LGT2 Indicates that a PR has LGTM 2. label Nov 4, 2020

Merge branch 'master' into explain-joins

8517d7d

TomShawn added the status/can-merge Indicates a PR has been approved by a committer. label Nov 4, 2020

TomShawn merged commit b108b9e into pingcap:master Nov 4, 2020

ti-srebot pushed a commit to ti-srebot/docs that referenced this pull request Nov 4, 2020

cherry pick pingcap#3922 to release-4.0

3fe5f74

Signed-off-by: ti-srebot <ti-srebot@pingcap.com>

ti-srebot mentioned this pull request Nov 4, 2020

explain: Add joins (#3922) #4147

Merged

9 tasks

TomShawn added the requires-followup This PR requires a follow-up task after being merged. label Nov 4, 2020

TomShawn pushed a commit that referenced this pull request Nov 4, 2020

cherry pick #3922 to release-4.0 (#4147)

68aa12e

Signed-off-by: ti-srebot <ti-srebot@pingcap.com> Co-authored-by: Null not nil <67764674+nullnotnil@users.noreply.github.com>

TomShawn assigned CharLotteiu and unassigned TomShawn Nov 4, 2020

ti-srebot mentioned this pull request Nov 9, 2020

explain: add subqueries (#3948) #4192

Merged

9 tasks

This was referenced Nov 13, 2020

toc: add sql optimization-related docs #4219

Merged

tikv config file: fix a typo pingcap/docs-cn#4906

Merged

TomShawn removed the requires-followup This PR requires a follow-up task after being merged. label Nov 16, 2020

CharLotteiu mentioned this pull request Nov 20, 2020

explain: add joins pingcap/docs-cn#4933

Merged

9 tasks

TomShawn added translation/done This PR has been translated from English into Chinese and updated to pingcap/docs-cn in a PR. and removed translation/doing This PR's assignee is translating this PR. labels Nov 23, 2020

This was referenced Jan 20, 2021

toc: add sql optimization-related docs (#4219) #4674

Merged

toc: add sql optimization-related docs (#4219) #4675

Merged

This was referenced Feb 20, 2021

explain: add joins (#4933) pingcap/docs-cn#5553

Merged

explain: add joins (#4933) pingcap/docs-cn#5554

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

explain: Add joins #3922

explain: Add joins #3922

ghost commented Sep 17, 2020 •

edited by ghost

ghost commented Sep 17, 2020 •

edited by ghost

ti-srebot commented Sep 20, 2020

SunRunAway commented Sep 21, 2020

ghost commented Sep 22, 2020

ti-srebot commented Sep 24, 2020

zz-jason Sep 25, 2020

ghost Sep 25, 2020

SunRunAway Sep 27, 2020

zz-jason Sep 29, 2020 •

edited

ghost Oct 1, 2020

TomShawn left a comment

ti-srebot commented Nov 4, 2020

ti-srebot commented Nov 4, 2020

ti-srebot commented Nov 4, 2020

TomShawn commented Nov 4, 2020


		### Runtime Statistics

		If [`tidb_mem_quota_query`](/system-variables.md#tidb_mem_quota_query) (default: 1GB) is exceeded, TiDB will attempt to use temporary storage provided that `oom-use-tmp-storage=TRUE` (default). This means that the hash table used as part of the hash join is created on disk. Runtime statistics, such as memory usage are visible in the `execution info` of `EXPLAIN ANALYZE`. The following example shows the output of `EXPLAIN ANALYZE` with a 1GB (default) `tidb_mem_quota_query` quota, and a 500MB quota. At 500MB the hash table spills to disk:

explain: Add joins #3922

explain: Add joins #3922

Conversation

ghost commented Sep 17, 2020 • edited by ghost

What is changed, added or deleted? (Required)

Which TiDB version(s) do your changes apply to? (Required)

What is the related PR or file link(s)?

Do your changes match any of the following descriptions?

ghost commented Sep 17, 2020 • edited by ghost

ti-srebot commented Sep 20, 2020

SunRunAway commented Sep 21, 2020

ghost commented Sep 22, 2020

ti-srebot commented Sep 24, 2020

zz-jason Sep 25, 2020

Choose a reason for hiding this comment

ghost Sep 25, 2020

Choose a reason for hiding this comment

SunRunAway Sep 27, 2020

Choose a reason for hiding this comment

zz-jason Sep 29, 2020 • edited

Choose a reason for hiding this comment

ghost Oct 1, 2020

Choose a reason for hiding this comment

TomShawn left a comment

Choose a reason for hiding this comment

ti-srebot commented Nov 4, 2020

ti-srebot commented Nov 4, 2020

ti-srebot commented Nov 4, 2020

TomShawn commented Nov 4, 2020

ghost commented Sep 17, 2020 •

edited by ghost

ghost commented Sep 17, 2020 •

edited by ghost

zz-jason Sep 29, 2020 •

edited