Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SPARK-13234] [SQL] remove duplicated SQL metrics #11163

Closed
wants to merge 3 commits into from

Conversation

davies
Copy link
Contributor

@davies davies commented Feb 10, 2016

For lots of SQL operators, we have metrics for both of input and output, the number of input rows should be exactly the number of output rows of child, we could only have metrics for output rows.

After we improved the performance using whole stage codegen, the overhead of SQL metrics are not trivial anymore, we should avoid that if it's not necessary.

This PR remove all the SQL metrics for number of input rows, add SQL metric of number of output rows for all LeafNode. All remove the SQL metrics from those operators that have the same number of rows from input and output (for example, Projection, we may don't need that).

The new SQL UI will looks like:

metrics

@andrewor14
Copy link
Contributor

@zsxwing

@SparkQA
Copy link

SparkQA commented Feb 10, 2016

Test build #51066 has finished for PR 11163 at commit 319f30d.

  • This patch fails Scala style tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@zsxwing
Copy link
Member

zsxwing commented Feb 11, 2016

LGTM

@SparkQA
Copy link

SparkQA commented Feb 11, 2016

Test build #51069 has finished for PR 11163 at commit 84af95d.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Feb 11, 2016

Test build #2535 has finished for PR 11163 at commit ecacbfc.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@davies
Copy link
Contributor Author

davies commented Feb 11, 2016

Merging into master, thanks!

@asfgit asfgit closed this in 8f744fe Feb 11, 2016
@SparkQA
Copy link

SparkQA commented Feb 11, 2016

Test build #51084 has finished for PR 11163 at commit ecacbfc.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants