[SPARK-1853] Show Streaming application code context (file, line number) in Spark Stages UI #2464

tdas · 2014-09-19T21:34:55Z

This is a refactored version of the original PR #1723 my @mubarak

Please take a look @andrewor14, @mubarak

…er) in Spark Stages UI

Conflicts: core/src/main/scala/org/apache/spark/rdd/RDD.scala streaming/src/main/scala/org/apache/spark/streaming/dstream/ReducedWindowedDStream.scala

…prevCallSite) only once. Adding return for other code paths (for None)

…ames in DStream

Conflicts: core/src/main/scala/org/apache/spark/util/Utils.scala

… unit test for DStream creation site.

…llsite Conflicts: streaming/src/main/scala/org/apache/spark/streaming/StreamingContext.scala streaming/src/test/scala/org/apache/spark/streaming/StreamingContextSuite.scala

SparkQA · 2014-09-19T21:39:35Z

QA tests have started for PR 2464 at commit 904cd92.

This patch merges cleanly.

tdas · 2014-09-19T21:43:16Z

streaming/src/main/scala/org/apache/spark/streaming/dstream/DStream.scala

-              None
+    // If RDD was already generated, then retrieve it from HashMap,
+    // or else compute the RDD
+    generatedRDDs.get(time).orElse {


These changes are just a refactoring of the code (from case Some and case None to Option.orElse), with no change in the logic.

I confirmed that this logic is the same as before

SparkQA · 2014-09-19T21:49:26Z

QA tests have started for PR 2464 at commit 390b45d.

This patch merges cleanly.

SparkQA · 2014-09-19T22:42:47Z

QA tests have finished for PR 2464 at commit 390b45d.

This patch fails unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2014-09-19T22:44:56Z

QA tests have finished for PR 2464 at commit 904cd92.

This patch passes unit tests.
This patch merges cleanly.
This patch adds no public classes.

andrewor14 · 2014-09-20T00:59:32Z

Thanks for working on this @tdas while on vacation. I will look at this in the next day or two.

mubarak · 2014-09-20T06:36:22Z

LGTM. Thanks @tdas

pwendell · 2014-09-21T17:47:47Z

core/src/main/scala/org/apache/spark/util/Utils.scala

   */
-  def getCallSite: CallSite = {
+  def getCallSite(classFilterFunc: String => Boolean = defaultCallSiteFilterFunc): CallSite = {


I would find it more natural to pass a function that excludes line from Spark code (it just inverts the logic). Something like internalExclusionFunc. Then you could have sparkCoreExclusionFunc and sparkStreamingExclusionFunc. @andrewor14 what do you find more intuitive?

Yeah, need a vote on this. I inverted the logic because the function has to distinguish not just Spark code but also ignore Scala classes (which may actually be in user code file). That;s why the default function has scala-regex inside it. So its best to treat this function as a user-code-or-not function rather than a spark-code-or-not function.

Yeah, I think negative is more intuitive than positive here. By default every line of the call stack is kept, but if the caller passes in his own ignoreClass function then we should respect that. I actually like the name skipClass more, because down there you can just do if (skipClass(el.getClassName)) { ... }

Alright, I am outvoted! But skipClass does not express that the parameter is a function. How about classExclusionFunc be the param name, and the actual functions be coreClassExclusionFunc and streamingClassExclusionFunc ?

SparkQA · 2014-09-23T20:34:56Z

QA tests have started for PR 2464 at commit dc54c71.

This patch merges cleanly.

SparkQA · 2014-09-23T21:45:39Z

QA tests have finished for PR 2464 at commit dc54c71.

This patch passes unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2014-09-23T21:45:42Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/20727/

andrewor14 · 2014-09-23T22:08:21Z

Thanks, LGTM. I'm merging this into master and 1.1

…er) in Spark Stages UI This is a refactored version of the original PR #1723 my mubarak Please take a look andrewor14, mubarak Author: Mubarak Seyed <mubarak.seyed@gmail.com> Author: Tathagata Das <tathagata.das1565@gmail.com> Closes #2464 from tdas/streaming-callsite and squashes the following commits: dc54c71 [Tathagata Das] Made changes based on PR comments. 390b45d [Tathagata Das] Fixed minor bugs. 904cd92 [Tathagata Das] Merge remote-tracking branch 'apache-github/master' into streaming-callsite 7baa427 [Tathagata Das] Refactored getCallSite and setCallSite to make it simpler. Also added unit test for DStream creation site. b9ed945 [Mubarak Seyed] Adding streaming utils c461cf4 [Mubarak Seyed] Merge remote-tracking branch 'upstream/master' ceb43da [Mubarak Seyed] Changing default regex function name 8c5d443 [Mubarak Seyed] Merge remote-tracking branch 'upstream/master' 196121b [Mubarak Seyed] Merge remote-tracking branch 'upstream/master' 491a1eb [Mubarak Seyed] Removing streaming visibility from getRDDCreationCallSite in DStream 33a7295 [Mubarak Seyed] Fixing review comments: Merging both setCallSite methods c26d933 [Mubarak Seyed] Merge remote-tracking branch 'upstream/master' f51fd9f [Mubarak Seyed] Fixing scalastyle, Regex for Utils.getCallSite, and changing method names in DStream 5051c58 [Mubarak Seyed] Getting return value of compute() into variable and call setCallSite(prevCallSite) only once. Adding return for other code paths (for None) a207eb7 [Mubarak Seyed] Fixing code review comments ccde038 [Mubarak Seyed] Removing Utils import from MappedDStream 2a09ad6 [Mubarak Seyed] Changes in Utils.scala for SPARK-1853 1d90cc3 [Mubarak Seyed] Changes for SPARK-1853 5f3105a [Mubarak Seyed] Merge remote-tracking branch 'upstream/master' 70f494f [Mubarak Seyed] Changes for SPARK-1853 1500deb [Mubarak Seyed] Changes in Spark Streaming UI 9d38d3c [Mubarak Seyed] [SPARK-1853] Show Streaming application code context (file, line number) in Spark Stages UI d466d75 [Mubarak Seyed] Changes for spark streaming UI (cherry picked from commit 729952a) Signed-off-by: Andrew Or <andrewor14@gmail.com>

tdas · 2014-09-24T06:11:32Z

@mubarak Thank you very much for this fix! Its finally merged!

mubarak and others added 21 commits July 17, 2014 18:15

Changes for spark streaming UI

d466d75

[SPARK-1853] Show Streaming application code context (file, line numb…

9d38d3c

…er) in Spark Stages UI

Changes in Spark Streaming UI

1500deb

Changes for SPARK-1853

70f494f

Merge remote-tracking branch 'upstream/master'

5f3105a

Conflicts: core/src/main/scala/org/apache/spark/rdd/RDD.scala streaming/src/main/scala/org/apache/spark/streaming/dstream/ReducedWindowedDStream.scala

Changes for SPARK-1853

1d90cc3

Changes in Utils.scala for SPARK-1853

2a09ad6

Removing Utils import from MappedDStream

ccde038

Fixing code review comments

a207eb7

Getting return value of compute() into variable and call setCallSite(…

5051c58

…prevCallSite) only once. Adding return for other code paths (for None)

Fixing scalastyle, Regex for Utils.getCallSite, and changing method n…

f51fd9f

…ames in DStream

Merge remote-tracking branch 'upstream/master'

c26d933

Fixing review comments: Merging both setCallSite methods

33a7295

Removing streaming visibility from getRDDCreationCallSite in DStream

491a1eb

Merge remote-tracking branch 'upstream/master'

196121b

Merge remote-tracking branch 'upstream/master'

8c5d443

Conflicts: core/src/main/scala/org/apache/spark/util/Utils.scala

Changing default regex function name

ceb43da

Merge remote-tracking branch 'upstream/master'

c461cf4

Adding streaming utils

b9ed945

Refactored getCallSite and setCallSite to make it simpler. Also added…

7baa427

… unit test for DStream creation site.

Merge remote-tracking branch 'apache-github/master' into streaming-ca…

904cd92

…llsite Conflicts: streaming/src/main/scala/org/apache/spark/streaming/StreamingContext.scala streaming/src/test/scala/org/apache/spark/streaming/StreamingContextSuite.scala

tdas mentioned this pull request Sep 19, 2014

[SPARK-1853] Show Streaming application code context (file, line number) in Spark Stages UI #1723

Closed

tdas reviewed Sep 19, 2014
View reviewed changes

Fixed minor bugs.

390b45d

pwendell reviewed Sep 21, 2014
View reviewed changes

Made changes based on PR comments.

dc54c71

asfgit closed this in 729952a Sep 23, 2014

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-1853] Show Streaming application code context (file, line number) in Spark Stages UI #2464

[SPARK-1853] Show Streaming application code context (file, line number) in Spark Stages UI #2464

tdas commented Sep 19, 2014

SparkQA commented Sep 19, 2014

tdas Sep 19, 2014

andrewor14 Sep 23, 2014

SparkQA commented Sep 19, 2014

SparkQA commented Sep 19, 2014

SparkQA commented Sep 19, 2014

andrewor14 commented Sep 20, 2014

mubarak commented Sep 20, 2014

pwendell Sep 21, 2014

tdas Sep 23, 2014

andrewor14 Sep 23, 2014

tdas Sep 23, 2014

SparkQA commented Sep 23, 2014

SparkQA commented Sep 23, 2014

SparkQA commented Sep 23, 2014

andrewor14 commented Sep 23, 2014

tdas commented Sep 24, 2014

[SPARK-1853] Show Streaming application code context (file, line number) in Spark Stages UI #2464

[SPARK-1853] Show Streaming application code context (file, line number) in Spark Stages UI #2464

Conversation

tdas commented Sep 19, 2014

SparkQA commented Sep 19, 2014

tdas Sep 19, 2014

Choose a reason for hiding this comment

andrewor14 Sep 23, 2014

Choose a reason for hiding this comment

SparkQA commented Sep 19, 2014

SparkQA commented Sep 19, 2014

SparkQA commented Sep 19, 2014

andrewor14 commented Sep 20, 2014

mubarak commented Sep 20, 2014

pwendell Sep 21, 2014

Choose a reason for hiding this comment

tdas Sep 23, 2014

Choose a reason for hiding this comment

andrewor14 Sep 23, 2014

Choose a reason for hiding this comment

tdas Sep 23, 2014

Choose a reason for hiding this comment

SparkQA commented Sep 23, 2014

SparkQA commented Sep 23, 2014

SparkQA commented Sep 23, 2014

andrewor14 commented Sep 23, 2014

tdas commented Sep 24, 2014