system-variables: add fine_grained_shuffle #9696
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
[REVIEW NOTIFICATION] This pull request has been approved by:
To complete the pull request process, please ask the reviewers in the list to review by filling The full list of commands accepted by this bot can be found here. Reviewer can indicate their review by submitting an approval review. |
ti-chi-bot
added
the
missing-translation-status
ti-chi-bot
added
the
size/S
hfxsd
added
v6.2
ti-chi-bot
added
the
size/S
guo-shaoge
reviewed
Jul 25, 2022
moyun
suggested changes
Jul 26, 2022
Co-authored-by: guo-shaoge <shaoge1994@163.com>
ran-huang
reviewed
Aug 1, 2022
Co-authored-by: Ran <huangran.alex@gmail.com>
Co-authored-by: Ran <huangran.alex@gmail.com>
|
/verify |
Co-authored-by: Ran <huangran.alex@gmail.com>
guo-shaoge
reviewed
Aug 4, 2022
system-variables.md
Outdated
| - Default value: `8192` | ||
| - Range: `[1, 18446744073709551616]` | ||
| - When Fine Grained Shuffle is enabled, the window function pushed down to TiFlash can be executed in parallel. This variable controls the batch size of the data sent by the sender. The sender will send data once the cumulative number of rows exceeds this value. | ||
| - Impact on performance: set a reasonable size according to your business requirements. Improper setting will affect the performance. If the value is set too small, for example `1`, it will cause one network transfer per Block. If the value is set too large, for example, the total number of rows of the table, it will cause the receiving end to spend most of the time waiting for data, and the piplelined computation can not work. You can observe the distribution of the number of rows received by the TiFlash receiver. If most threads receive only a few rows, for example a few hundred, you can increase this value to reduce the network overhead. |
There was a problem hiding this comment.
Suggested change
| - Impact on performance: set a reasonable size according to your business requirements. Improper setting will affect the performance. If the value is set too small, for example `1`, it will cause one network transfer per Block. If the value is set too large, for example, the total number of rows of the table, it will cause the receiving end to spend most of the time waiting for data, and the piplelined computation can not work. You can observe the distribution of the number of rows received by the TiFlash receiver. If most threads receive only a few rows, for example a few hundred, you can increase this value to reduce the network overhead. | |
| - Impact on performance: If the value is set too small, for example `1`, it will cause one network transfer per Block. If the value is set too large, for example, the total number of rows of the table, it will cause the receiving end to spend most of the time waiting for data, and the piplelined computation can not work. You can observe the distribution of the number of rows received by the TiFlash receiver. If most threads receive only a few rows, for example a few hundred, you can increase this value to reduce the network overhead. |
There was a problem hiding this comment.
Let's keep this sentence.
Co-authored-by: guo-shaoge <shaoge1994@163.com>
ran-huang
approved these changes
Aug 4, 2022
guo-shaoge
approved these changes
Aug 4, 2022
moyun
approved these changes
Aug 4, 2022
|
/merge |
|
@hfxsd: In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository. |
|
/merge |
|
This pull request has been accepted and is ready to merge. Commit hash: bf7ac44
|
ti-chi-bot
added
the
status/can-merge
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What is changed, added or deleted? (Required)
Added two variables:
tiflash_fine_grained_shuffle_batch_sizetiflash_fine_grained_shuffle_stream_countWhich TiDB version(s) do your changes apply to? (Required)
Tips for choosing the affected version(s):
By default, CHOOSE MASTER ONLY so your changes will be applied to the next TiDB major or minor releases. If your PR involves a product feature behavior change or a compatibility change, CHOOSE THE AFFECTED RELEASE BRANCH(ES) AND MASTER.
For details, see tips for choosing the affected versions.
What is the related PR or file link(s)?
Do your changes match any of the following descriptions?