[SPARK-35529][SQL] Add fallback metrics for hash aggregate #32671
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
@cloud-fan could you help take a look when you have time? Thanks. |
cloud-fan
reviewed
May 26, 2021
| SQLMetrics.createAverageMetric(sparkContext, "avg hash probe bucket list iters")) | ||
| SQLMetrics.createAverageMetric(sparkContext, "avg hash probe bucket list iters"), | ||
| "numTasksFallBacked" -> SQLMetrics.createMetric(sparkContext, | ||
| "number of tasks fall-backed to sort-based aggregation")) |
There was a problem hiding this comment.
is this name the same with object hash agg? it is super long..
probably "number of sort fallback tasks"
There was a problem hiding this comment.
Yes, same as in #31340. Let me change them together.
cloud-fan
approved these changes
May 26, 2021
|
Kubernetes integration test starting |
|
Kubernetes integration test status success |
|
thanks, merging to master! |
|
Test build #138976 has finished for PR 32671 at commit
|
|
Test build #138972 has finished for PR 32671 at commit
|
|
Thank you @cloud-fan for review! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Add the metrics to record how many tasks fallback to sort-based aggregation for hash aggregation. This will help developers and users to debug and optimize query. Object hash aggregation has similar metrics already.
Why are the changes needed?
Help developers and users to debug and optimize query with hash aggregation.
Does this PR introduce any user-facing change?
Yes, the added metrics will show up in Spark web UI.
Example:
How was this patch tested?
Changed unit test in
SQLMetricsSuite.scala.