Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SPARK-2950] Add gc time and shuffle write time to JobLogger #1869

Closed
wants to merge 2 commits into from
Closed

[SPARK-2950] Add gc time and shuffle write time to JobLogger #1869

wants to merge 2 commits into from

Conversation

shivaram
Copy link
Contributor

@shivaram shivaram commented Aug 9, 2014

The JobLogger is very useful for performing offline performance profiling of Spark jobs. GC Time and Shuffle Write time are available in TaskMetrics but are currently missed from the JobLogger output. This patch adds these two fields.

Since this is a small change, I didn't create a JIRA. Let me know if I should do that.

cc @kayousterhout

@SparkQA
Copy link

SparkQA commented Aug 9, 2014

QA tests have started for PR 1869. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18264/consoleFull

@SparkQA
Copy link

SparkQA commented Aug 9, 2014

QA results for PR 1869:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18264/consoleFull

@aarondav
Copy link
Contributor

It's a small thing, but it'd be nice to have a JIRA in case someone asks for this feature, we can link the JIRA and it shows which version the fix will appear in.

@shivaram shivaram changed the title Add gc time and shuffle write time to JobLogger [SPARK-2950] Add gc time and shuffle write time to JobLogger Aug 10, 2014
@shivaram
Copy link
Contributor Author

Created a JIRA and updated the title.

@SparkQA
Copy link

SparkQA commented Aug 10, 2014

QA tests have started for PR 1869. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18265/consoleFull

@SparkQA
Copy link

SparkQA commented Aug 10, 2014

QA results for PR 1869:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18265/consoleFull

@kayousterhout
Copy link
Contributor

Looks great!! +1 on this being useful.

@shivaram
Copy link
Contributor Author

Merged this into master. @pwendell Could we cherry pick this for 1.1 as well ? Its a small change and will allow profiling clusters running 1.1

@asfgit asfgit closed this in 1d03a26 Aug 10, 2014
xiliu82 pushed a commit to xiliu82/spark that referenced this pull request Sep 4, 2014
@shivaram @conviva-zz
[SPARK-2950] Add gc time and shuffle write time to JobLogger
67b584d 
The JobLogger is very useful for performing offline performance profiling of Spark jobs. GC Time and Shuffle Write time are available in TaskMetrics but are currently missed from the JobLogger output. This patch adds these two fields.

~~Since this is a small change, I didn't create a JIRA. Let me know if I should do that.~~

cc kayousterhout

Author: Shivaram Venkataraman <shivaram@cs.berkeley.edu>

Closes apache#1869 from shivaram/job-logger and squashes the following commits:

1b709fc [Shivaram Venkataraman] Add a space before GC_TIME
c418105 [Shivaram Venkataraman] Add gc time and shuffle write time to JobLogger
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
4 participants
@shivaram @SparkQA @aarondav @kayousterhout