SPARK-7171: Added a method to retrieve metrics sources in TaskContext #5805

jacek-lewandowski · 2015-04-30T08:31:50Z

No description provided.

SparkQA · 2015-04-30T09:07:14Z

Test build #31406 has finished for PR 5805 at commit 92aa76f.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds no public classes.
This patch adds the following new dependencies:
- jaxb-api-2.2.7.jar
- jaxb-core-2.2.7.jar
- jaxb-impl-2.2.7.jar
- pmml-agent-1.1.15.jar
- pmml-model-1.1.15.jar
- pmml-schema-1.1.15.jar
This patch removes the following dependencies:
- activation-1.1.jar
- jaxb-api-2.2.2.jar
- jaxb-impl-2.2.3-1.jar

jacek-lewandowski · 2015-04-30T09:09:17Z

retest this please

SparkQA · 2015-04-30T10:53:56Z

Test build #31412 has finished for PR 5805 at commit 9ed86ca.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

squito · 2015-04-30T20:48:21Z

core/src/main/scala/org/apache/spark/TaskContextImpl.scala

@@ -94,5 +97,8 @@ private[spark] class TaskContextImpl(
  override def isRunningLocally(): Boolean = runningLocally

  override def isInterrupted(): Boolean = interrupted
+
+  override def getMetricsSources(sourceName: String): Seq[Source] =
+    metricsSystem.getSourcesByName(sourceName)


will you get an NPE here if no metric system has been setup?

nevermind, sorry -- I see that its never null, it can just be empty

squito · 2015-04-30T20:51:31Z

Hi @jacek-lewandowski
I like this idea overall, but personally I am little hesitant to add something like this in the flurry before 1.4 is released since its changing public apis (even though its a non-breaking change). Not that I am against it if somebody else feels safe merging this -- just wanted to give you a heads up that you might need to sit on it a while.

jacek-lewandowski · 2015-05-04T05:27:33Z

@squito so who should I ask for this?

andrewor14 · 2015-06-18T23:42:41Z

core/src/main/scala/org/apache/spark/TaskContext.scala

@@ -135,6 +136,10 @@ abstract class TaskContext extends Serializable {
  @DeveloperApi
  def taskMetrics(): TaskMetrics

+  /** ::DeveloperApi:: */


This special tag is only necessary if there is a java doc (please add one). The javadoc here will be incorrectly displayed on the docs page. For more info see other places where :: DeveloperApi :: is used.

andrewor14 · 2015-06-18T23:43:15Z

@jacek-lewandowski once you rebase to master I'll take a look at this in more detail.

jacek-lewandowski · 2015-06-22T06:33:02Z

@andrewor14 rebased

SparkQA · 2015-06-22T09:13:59Z

Test build #35435 has finished for PR 5805 at commit 3d06806.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-06-29T20:48:12Z

Test build #36031 has finished for PR 5805 at commit 3d06806.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

andrewor14 · 2015-07-17T21:22:58Z

@jacek-lewandowski I just looked at this and the changes look fine. Unfortunately it doesn't merge with master again. Could you rebase this once more? Also, I'd like to see my documentation comment addressed. Once you've done these things then we can probably merge this.

SparkQA · 2015-07-20T15:46:09Z

Test build #37826 has finished for PR 5805 at commit d685e53.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class ShuffledRowRDD(

SparkQA · 2015-07-20T16:42:27Z

Test build #37844 has finished for PR 5805 at commit 9837a67.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-07-20T20:00:41Z

Test build #37852 has finished for PR 5805 at commit ed20bda.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

jacek-lewandowski · 2015-07-20T20:19:57Z

@andrewor14 is it ok now?

andrewor14 · 2015-07-21T16:52:57Z

LGTM, merging into master. Thanks @jacek-lewandowski.

jacek-lewandowski force-pushed the SPARK-7171 branch from 92aa76f to 9ed86ca Compare April 30, 2015 09:08

squito reviewed Apr 30, 2015
View reviewed changes

andrewor14 reviewed Jun 18, 2015
View reviewed changes

jacek-lewandowski force-pushed the SPARK-7171 branch from 9ed86ca to 3d06806 Compare June 22, 2015 06:32

jacek-lewandowski force-pushed the SPARK-7171 branch from 3d06806 to d685e53 Compare July 20, 2015 08:58

jacek-lewandowski force-pushed the SPARK-7171 branch from d685e53 to 9837a67 Compare July 20, 2015 16:24

SPARK-7171: Added a method to retrieve metrics sources in TaskContext

ed20bda

jacek-lewandowski force-pushed the SPARK-7171 branch from 9837a67 to ed20bda Compare July 20, 2015 17:43

asfgit closed this in 3195491 Jul 21, 2015

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

SPARK-7171: Added a method to retrieve metrics sources in TaskContext #5805

SPARK-7171: Added a method to retrieve metrics sources in TaskContext #5805

jacek-lewandowski commented Apr 30, 2015

SparkQA commented Apr 30, 2015

jacek-lewandowski commented Apr 30, 2015

SparkQA commented Apr 30, 2015

squito Apr 30, 2015

squito Apr 30, 2015

squito commented Apr 30, 2015

jacek-lewandowski commented May 4, 2015

andrewor14 Jun 18, 2015

andrewor14 commented Jun 18, 2015

jacek-lewandowski commented Jun 22, 2015

SparkQA commented Jun 22, 2015

SparkQA commented Jun 29, 2015

andrewor14 commented Jul 17, 2015

SparkQA commented Jul 20, 2015

SparkQA commented Jul 20, 2015

SparkQA commented Jul 20, 2015

jacek-lewandowski commented Jul 20, 2015

andrewor14 commented Jul 21, 2015

SPARK-7171: Added a method to retrieve metrics sources in TaskContext #5805

SPARK-7171: Added a method to retrieve metrics sources in TaskContext #5805

Conversation

jacek-lewandowski commented Apr 30, 2015

SparkQA commented Apr 30, 2015

jacek-lewandowski commented Apr 30, 2015

SparkQA commented Apr 30, 2015

squito Apr 30, 2015

Choose a reason for hiding this comment

squito Apr 30, 2015

Choose a reason for hiding this comment

squito commented Apr 30, 2015

jacek-lewandowski commented May 4, 2015

andrewor14 Jun 18, 2015

Choose a reason for hiding this comment

andrewor14 commented Jun 18, 2015

jacek-lewandowski commented Jun 22, 2015

SparkQA commented Jun 22, 2015

SparkQA commented Jun 29, 2015

andrewor14 commented Jul 17, 2015

SparkQA commented Jul 20, 2015

SparkQA commented Jul 20, 2015

SparkQA commented Jul 20, 2015

jacek-lewandowski commented Jul 20, 2015

andrewor14 commented Jul 21, 2015