SPARK-3874: Provide stable TaskContext API #2803

pwendell · 2014-10-14T15:43:16Z

This is a small number of clean-up changes on top of #2782. Closes #2782.

pwendell · 2014-10-14T15:43:56Z

Jenkins, test this please. With these changes this LGTM.

SparkQA · 2014-10-14T16:03:48Z

QA tests have started for PR 2803 at commit e10c08d.

This patch merges cleanly.

SparkQA · 2014-10-14T16:29:35Z

QA tests have started for PR 2803 at commit 44089ec.

This patch merges cleanly.

SparkQA · 2014-10-14T17:14:44Z

QA tests have finished for PR 2803 at commit e10c08d.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- public abstract class TaskContext implements Serializable

AmplabJenkins · 2014-10-14T17:14:50Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21730/
Test FAILed.

SparkQA · 2014-10-14T17:59:08Z

QA tests have finished for PR 2803 at commit 44089ec.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- public abstract class TaskContext implements Serializable

AmplabJenkins · 2014-10-14T17:59:13Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21731/
Test PASSed.

JoshRosen · 2014-10-14T18:20:38Z

project/MimaBuild.scala

@@ -40,6 +40,7 @@ object MimaBuild {
      ProblemFilters.exclude[IncompatibleResultTypeProblem](fullName),
      ProblemFilters.exclude[IncompatibleMethTypeProblem](fullName),
      ProblemFilters.exclude[IncompatibleFieldTypeProblem](fullName)
+//      ProblemFilters.exclude[AbstractClassProblem](fullName)


Could we just remove this instead of commenting it out?

JoshRosen · 2014-10-14T18:23:36Z

LGTM.

vanzin · 2014-10-14T20:55:48Z

core/src/main/java/org/apache/spark/TaskContext.java

-  }
+  public abstract boolean isInterrupted();
+
+  @Deprecated


super nit: add annotation after javadoc.

pwendell · 2014-10-14T23:04:26Z

Thanks @JoshRosen and @vanzin - I've pushed a patch to address the style issues.

AmplabJenkins · 2014-10-14T23:52:18Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21746/
Test FAILed.

pwendell · 2014-10-15T05:09:27Z

Jenkins, retest this please.

AmplabJenkins · 2014-10-15T05:57:20Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21761/
Test FAILed.

SparkQA · 2014-10-15T06:06:28Z

QA tests have started for PR 2803 at commit 56d5b7a.

This patch merges cleanly.

ScrapCodes · 2014-10-15T06:28:16Z

project/MimaExcludes.scala

+              "org.apache.spark.scheduler.MapStatus"),
+            // TaskContext was promoted to Abstract class
+            ProblemFilters.exclude[AbstractClassProblem](
+              "org.apache.spark.TaskContext")


I felt we could exclude this generally, it did not made sense to me to report these errors for classes marked developerApi or experimental. Thoughts ?

SparkQA · 2014-10-15T07:17:01Z

QA tests have finished for PR 2803 at commit 56d5b7a.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- public abstract class TaskContext implements Serializable

pwendell · 2014-10-16T03:26:19Z

Jenkins, test this please.

SparkQA · 2014-10-16T03:29:42Z

QA tests have started for PR 2803 at commit 56d5b7a.

This patch merges cleanly.

SparkQA · 2014-10-16T04:41:23Z

QA tests have finished for PR 2803 at commit 56d5b7a.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- public abstract class TaskContext implements Serializable

AmplabJenkins · 2014-10-16T04:41:27Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21788/
Test FAILed.

pwendell · 2014-10-16T06:00:13Z

Jenkins, test this please. @liancheng - I am getting a SQL failure here that seems like it might be legitimate but I'm having trouble interpreting why it could possibly be caused by this patch - any idea?

pwendell · 2014-10-16T06:00:26Z

Jenkins, test this please.

SparkQA · 2014-10-16T06:04:41Z

QA tests have started for PR 2803 at commit 56d5b7a.

This patch merges cleanly.

SparkQA · 2014-10-16T07:14:57Z

QA tests have finished for PR 2803 at commit 56d5b7a.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- public abstract class TaskContext implements Serializable

AmplabJenkins · 2014-10-16T07:15:01Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21794/
Test FAILed.

pwendell · 2014-10-16T23:26:58Z

Jenkins, test this please.

SparkQA · 2014-10-16T23:29:55Z

QA tests have started for PR 2803 at commit 56d5b7a.

This patch merges cleanly.

SparkQA · 2014-10-17T00:59:23Z

QA tests have finished for PR 2803 at commit 56d5b7a.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- public abstract class TaskContext implements Serializable

AmplabJenkins · 2014-10-17T00:59:27Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21826/
Test PASSed.

pwendell · 2014-10-17T01:37:57Z

Great @liancheng that seems to have done it.

liancheng · 2014-10-17T02:36:11Z

@pwendell I'm not pretty sure about that... This PR fixes the race condition, but the first Jenkins build of this PR also suffered the spark.local.dir clean up failure issue, which doesn't happen deterministically. It seems that the temporary directories under /tmp got corrupted. Maybe it's related to Jenkins environment. I observed this kind of failure pretty frequently last night (e.g. build 21797), but now it's gone. Let's wait and see whether it happens again.

ScrapCodes added 6 commits October 13, 2014 18:13

SPARK-3874, Provide stable TaskContext API

ef633f5

adding missed out files to git.

bbd9e05

CR, Moved implementations to TaskContextImpl

7ecc2fe

Fixed the mima issue.

facf3b1

Josh's suggestion

df261d0

Fixed a typo

ed551ce

Clean-up the TaskContext API.

44089ec

pwendell force-pushed the pr-2782 branch from e10c08d to 44089ec Compare October 14, 2014 16:25

JoshRosen reviewed Oct 14, 2014
View reviewed changes

vanzin reviewed Oct 14, 2014
View reviewed changes

Minor clean-up

56d5b7a

ScrapCodes reviewed Oct 15, 2014
View reviewed changes

asfgit closed this in 2fe0ba9 Oct 17, 2014

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

SPARK-3874: Provide stable TaskContext API #2803

SPARK-3874: Provide stable TaskContext API #2803

pwendell commented Oct 14, 2014

pwendell commented Oct 14, 2014

SparkQA commented Oct 14, 2014

SparkQA commented Oct 14, 2014

SparkQA commented Oct 14, 2014

AmplabJenkins commented Oct 14, 2014

SparkQA commented Oct 14, 2014

AmplabJenkins commented Oct 14, 2014

JoshRosen Oct 14, 2014

JoshRosen commented Oct 14, 2014

vanzin Oct 14, 2014

pwendell commented Oct 14, 2014

AmplabJenkins commented Oct 14, 2014

pwendell commented Oct 15, 2014

AmplabJenkins commented Oct 15, 2014

SparkQA commented Oct 15, 2014

ScrapCodes Oct 15, 2014

SparkQA commented Oct 15, 2014

pwendell commented Oct 16, 2014

SparkQA commented Oct 16, 2014

SparkQA commented Oct 16, 2014

AmplabJenkins commented Oct 16, 2014

pwendell commented Oct 16, 2014

pwendell commented Oct 16, 2014

SparkQA commented Oct 16, 2014

SparkQA commented Oct 16, 2014

AmplabJenkins commented Oct 16, 2014

pwendell commented Oct 16, 2014

SparkQA commented Oct 16, 2014

SparkQA commented Oct 17, 2014

AmplabJenkins commented Oct 17, 2014

pwendell commented Oct 17, 2014

liancheng commented Oct 17, 2014

SPARK-3874: Provide stable TaskContext API #2803

SPARK-3874: Provide stable TaskContext API #2803

Conversation

pwendell commented Oct 14, 2014

pwendell commented Oct 14, 2014

SparkQA commented Oct 14, 2014

SparkQA commented Oct 14, 2014

SparkQA commented Oct 14, 2014

AmplabJenkins commented Oct 14, 2014

SparkQA commented Oct 14, 2014

AmplabJenkins commented Oct 14, 2014

JoshRosen Oct 14, 2014

Choose a reason for hiding this comment

JoshRosen commented Oct 14, 2014

vanzin Oct 14, 2014

Choose a reason for hiding this comment

pwendell commented Oct 14, 2014

AmplabJenkins commented Oct 14, 2014

pwendell commented Oct 15, 2014

AmplabJenkins commented Oct 15, 2014

SparkQA commented Oct 15, 2014

ScrapCodes Oct 15, 2014

Choose a reason for hiding this comment

SparkQA commented Oct 15, 2014

pwendell commented Oct 16, 2014

SparkQA commented Oct 16, 2014

SparkQA commented Oct 16, 2014

AmplabJenkins commented Oct 16, 2014

pwendell commented Oct 16, 2014

pwendell commented Oct 16, 2014

SparkQA commented Oct 16, 2014

SparkQA commented Oct 16, 2014

AmplabJenkins commented Oct 16, 2014

pwendell commented Oct 16, 2014

SparkQA commented Oct 16, 2014

SparkQA commented Oct 17, 2014

AmplabJenkins commented Oct 17, 2014

pwendell commented Oct 17, 2014

liancheng commented Oct 17, 2014