[SPARK-28935][SQL][DOCS] Document SQL metrics for Details for Query Plan #25658
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Test build #110033 has finished for PR 25658 at commit
|
dilipbiswal
reviewed
Sep 3, 2019
docs/web-ui.md
Outdated
| The metrics of SQL operators show in the block of operators. The SQL metrics can be useful when | ||
| we want to dive into the execution details of each operator, for example, how many rows are output | ||
| after a Filter operator. The related metrics are different for each type of operator, for example | ||
| Exchange has the metrics called "shuffle bytes writte total" which shows the number of bytes written |
dilipbiswal
reviewed
Sep 3, 2019
docs/web-ui.md
Outdated
| <tr><td> <code>metadata time</code> </td><td> the time spent on getting metadata like number of partitions, number of files </td><td> FileSourceScan </td></tr> | ||
| <tr><td> <code>shuffle bytes written</code> </td><td> number of bytes written </td><td> CollectLimit, TakeOrderedAndProject, ShuffleExchange </td></tr> | ||
| <tr><td> <code>shuffle records written</code> </td><td> number of records written </td><td> CollectLimit, TakeOrderedAndProject, ShuffleExchange </td></tr> | ||
| <tr><td> <code>shuffle write time</code> </td><td> the time on shuffle writing </td><td> CollectLimit, TakeOrderedAndProject, ShuffleExchange </td></tr> |
There was a problem hiding this comment.
Nit: the time spent on writing shuffle data ?
dilipbiswal
reviewed
Sep 3, 2019
docs/web-ui.md
Outdated
| @@ -363,6 +363,42 @@ number of written shuffle records, total data size, etc. | |||
| Clicking the 'Details' link on the bottom displays the logical plans and the physical plan, which | |||
| illustrate how Spark parses, analyzes, optimizes and performs the query. | |||
|
|
|||
| ### SQL metrics | |||
|
|
|||
| The metrics of SQL operators show in the block of operators. The SQL metrics can be useful when | |||
There was a problem hiding this comment.
@viirya a little confused on "show in the block of operators" ? Is there a way to reword this ?
|
@viirya Looks good to me. I have some minor comments :-) |
|
LGTM after a few minor updates! |
|
thanks for updating! @gatorsmile |
|
Test build #110263 has finished for PR 25658 at commit
|
|
Thanks! Merged to master. |
PavithraRamachandran
pushed a commit
to PavithraRamachandran/spark
that referenced
this pull request
Sep 15, 2019
### What changes were proposed in this pull request? This patch adds the description of common SQL metrics in web ui document. ### Why are the changes needed? The current web ui document describes query plan but does not describe the meaning SQL metrics. For end users, they might not understand the meaning of the metrics. ### Does this PR introduce any user-facing change? No. This is just documentation change. ### How was this patch tested? Built the docs locally.  Closes apache#25658 from viirya/SPARK-28935. Lead-authored-by: Liang-Chi Hsieh <viirya@gmail.com> Co-authored-by: Xiao Li <gatorsmile@gmail.com> Signed-off-by: Xiao Li <gatorsmile@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
This patch adds the description of common SQL metrics in web ui document.
Why are the changes needed?
The current web ui document describes query plan but does not describe the meaning SQL metrics. For end users, they might not understand the meaning of the metrics.
Does this PR introduce any user-facing change?
No. This is just documentation change.
How was this patch tested?
Built the docs locally.