SPARK-3179. Add task OutputMetrics. #2968

sryza · 2014-10-28T00:55:26Z

No description provided.

SparkQA · 2014-10-28T00:59:44Z

Test build #22314 has started for PR 2968 at commit 1a793e8.

This patch merges cleanly.

SparkQA · 2014-10-28T02:10:02Z

Test build #22314 has finished for PR 2968 at commit 1a793e8.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class OutputMetrics()

AmplabJenkins · 2014-10-28T02:10:06Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22314/
Test PASSed.

sryza · 2014-10-31T22:51:09Z

Updated patch rebases on master and adds output metrics to a couple places on the UI they were missing.

sryza · 2014-10-31T22:51:35Z

Verified that output metrics show up every (and while tasks are running) on a pseudo-distributed cluster.

SparkQA · 2014-10-31T22:55:01Z

Test build #22657 has started for PR 2968 at commit 94b7c3f.

This patch merges cleanly.

SparkQA · 2014-11-01T00:08:30Z

Test build #22657 has finished for PR 2968 at commit 94b7c3f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-11-01T00:08:33Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22657/
Test PASSed.

kayousterhout · 2014-11-06T07:32:25Z

core/src/main/scala/org/apache/spark/executor/TaskMetrics.scala

+  /**
+   * Total bytes written
+   */
+  var bytesWritten: Long = 0L


Do you anticipate adding other things here later (like time blocked on writing output data)? Or should this just be a long inside the metrics class (rather than having those whole output metrics class)?

kayousterhout · 2014-11-06T07:44:46Z

@sryza this mostly looks good and super helpful!! One more comment I discussed with @pwendell: I think it would be good to add an output type field to output metrics (similar to what we have for input metrics) that currently can only be Hadoop. The motivation for this is that there's some annoying overhead associated with changing stuff we write out to Json, and it seems like we're likely to want to add metrics for other output types (e.g., storing output data in the block manager) later on.

sryza · 2014-11-06T08:21:31Z

Thanks for taking a look @kayousterhout . I'll add in an output type.

SparkQA · 2014-11-06T09:22:40Z

Test build #22997 has started for PR 2968 at commit 92c772c.

This patch merges cleanly.

SparkQA · 2014-11-06T10:27:25Z

Test build #22997 has finished for PR 2968 at commit 92c772c.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class OutputMetrics(writeMethod: DataWriteMethod.Value)

AmplabJenkins · 2014-11-06T10:27:28Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22997/
Test FAILed.

kayousterhout · 2014-11-06T21:51:08Z

Jenkins, retest this please

SparkQA · 2014-11-06T21:55:02Z

Test build #23018 has started for PR 2968 at commit 92c772c.

This patch merges cleanly.

SparkQA · 2014-11-06T23:18:57Z

Test build #23018 has finished for PR 2968 at commit 92c772c.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class OutputMetrics(writeMethod: DataWriteMethod.Value)

AmplabJenkins · 2014-11-06T23:19:01Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23018/
Test PASSed.

kayousterhout · 2014-11-08T00:57:28Z

core/src/main/scala/org/apache/spark/deploy/SparkHadoopUtil.scala

+    : Option[() => Long] = {
+    try {
+      val threadStats = getFileSystemThreadStatistics(path, conf)
+      val getBytesReadMethod = getFileSystemThreadStatisticsMethod("getBytesWritten")


getBytesWrittenMethod? And same with the below naming?

kayousterhout · 2014-11-08T01:16:04Z

Sorry for not noticing the two other things earlier!!

kayousterhout · 2014-11-08T01:19:30Z

core/src/test/scala/org/apache/spark/metrics/InputOutputMetricsSuite.scala

+    val fs = FileSystem.getLocal(new Configuration())
+    val outPath = new Path(fs.getWorkingDirectory, "outdir")
+
+    if (SparkHadoopUtil.get.getFSBytesWrittenOnThreadCallback(outPath, fs.getConf).isDefined) {


This test is a little hairy because it could break without anyone noticing (since I don't think Jenkins runs a new enough version for this code to be run?). But I don't have a great solution -- you could mock out a bunch of the function calls, but then you're left not really testing anything.

Yeah, I had this same thought, but I can't think of any good way to prevent this. Note that the other tests in this class have the same problem.

SparkQA · 2014-11-08T23:17:37Z

Test build #23107 has started for PR 2968 at commit dce4784.

This patch merges cleanly.

SparkQA · 2014-11-09T00:44:26Z

Test build #23107 has finished for PR 2968 at commit dce4784.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class OutputMetrics(writeMethod: DataWriteMethod.Value)

AmplabJenkins · 2014-11-09T00:44:30Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23107/
Test PASSed.

kayousterhout · 2014-11-10T06:32:55Z

Thanks Sandy! I've merged this into master and 1.2.

Author: Sandy Ryza <sandy@cloudera.com> This patch had conflicts when merged, resolved by Committer: Kay Ousterhout <kayousterhout@gmail.com> Closes #2968 from sryza/sandy-spark-3179 and squashes the following commits: dce4784 [Sandy Ryza] More review feedback 8d350d1 [Sandy Ryza] Fix test against Hadoop 2.5+ e7c74d0 [Sandy Ryza] More review feedback 6cff9c4 [Sandy Ryza] Review feedback fb2dde0 [Sandy Ryza] SPARK-3179 (cherry picked from commit 3c2cff4) Signed-off-by: Kay Ousterhout <kayousterhout@gmail.com>

sryza force-pushed the sandy-spark-3179 branch from 1a793e8 to 94b7c3f Compare October 31, 2014 22:49

kayousterhout reviewed Nov 6, 2014
View reviewed changes

sryza force-pushed the sandy-spark-3179 branch from 94b7c3f to 92c772c Compare November 6, 2014 09:18

kayousterhout reviewed Nov 8, 2014
View reviewed changes

sryza added 4 commits November 8, 2014 14:58

SPARK-3179

fb2dde0

Review feedback

6cff9c4

More review feedback

e7c74d0

Fix test against Hadoop 2.5+

8d350d1

More review feedback

dce4784

sryza force-pushed the sandy-spark-3179 branch from 92c772c to dce4784 Compare November 8, 2014 23:12

asfgit closed this in 3c2cff4 Nov 10, 2014

pwendell mentioned this pull request Feb 5, 2015

[SPARK-4874] [CORE] Collect record count metrics #4067

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

SPARK-3179. Add task OutputMetrics. #2968

SPARK-3179. Add task OutputMetrics. #2968

sryza commented Oct 28, 2014

SparkQA commented Oct 28, 2014

SparkQA commented Oct 28, 2014

AmplabJenkins commented Oct 28, 2014

sryza commented Oct 31, 2014

sryza commented Oct 31, 2014

SparkQA commented Oct 31, 2014

SparkQA commented Nov 1, 2014

AmplabJenkins commented Nov 1, 2014

kayousterhout Nov 6, 2014

kayousterhout commented Nov 6, 2014

sryza commented Nov 6, 2014

SparkQA commented Nov 6, 2014

SparkQA commented Nov 6, 2014

AmplabJenkins commented Nov 6, 2014

kayousterhout commented Nov 6, 2014

SparkQA commented Nov 6, 2014

SparkQA commented Nov 6, 2014

AmplabJenkins commented Nov 6, 2014

kayousterhout Nov 8, 2014

kayousterhout commented Nov 8, 2014

kayousterhout Nov 8, 2014

sryza Nov 8, 2014

SparkQA commented Nov 8, 2014

SparkQA commented Nov 9, 2014

AmplabJenkins commented Nov 9, 2014

kayousterhout commented Nov 10, 2014

SPARK-3179. Add task OutputMetrics. #2968

SPARK-3179. Add task OutputMetrics. #2968

Conversation

sryza commented Oct 28, 2014

SparkQA commented Oct 28, 2014

SparkQA commented Oct 28, 2014

AmplabJenkins commented Oct 28, 2014

sryza commented Oct 31, 2014

sryza commented Oct 31, 2014

SparkQA commented Oct 31, 2014

SparkQA commented Nov 1, 2014

AmplabJenkins commented Nov 1, 2014

kayousterhout Nov 6, 2014

Choose a reason for hiding this comment

kayousterhout commented Nov 6, 2014

sryza commented Nov 6, 2014

SparkQA commented Nov 6, 2014

SparkQA commented Nov 6, 2014

AmplabJenkins commented Nov 6, 2014

kayousterhout commented Nov 6, 2014

SparkQA commented Nov 6, 2014

SparkQA commented Nov 6, 2014

AmplabJenkins commented Nov 6, 2014

kayousterhout Nov 8, 2014

Choose a reason for hiding this comment

kayousterhout commented Nov 8, 2014

kayousterhout Nov 8, 2014

Choose a reason for hiding this comment

sryza Nov 8, 2014

Choose a reason for hiding this comment

SparkQA commented Nov 8, 2014

SparkQA commented Nov 9, 2014

AmplabJenkins commented Nov 9, 2014

kayousterhout commented Nov 10, 2014