[SPARK-25248] [CORE] Audit barrier Scala APIs for 2.4 #22240

mengxr · 2018-08-27T04:22:21Z

What changes were proposed in this pull request?

I made one pass over barrier APIs added to Spark 2.4 and updates some scopes and docs. I will update Python docs once Scala doc was reviewed.

One major issue is that BarrierTaskContext implements TaskContextImpl that exposes some public methods. And internally there were several direct references to TaskContextImpl methods instead of TaskContext. This PR moved some methods from TaskContextImpl to TaskContext, remaining package private, and used delegate methods to avoid inheriting TaskContextImp and exposing unnecessary APIs.

TODOs:

scala doc
python doc ([SPARK-25248.1][PYSPARK] update barrier Python API #22261 ).

mengxr · 2018-08-27T04:23:12Z

core/src/main/scala/org/apache/spark/BarrierTaskContext.scala

    override val stageId: Int,
    override val stageAttemptNumber: Int,
    override val partitionId: Int,
    override val taskAttemptId: Long,
    override val attemptNumber: Int,
-    override val taskMemoryManager: TaskMemoryManager,
+    private[spark] override val taskMemoryManager: TaskMemoryManager,


This is not exposed by TaskContext.

mengxr · 2018-08-27T04:23:51Z

core/src/main/scala/org/apache/spark/BarrierTaskContext.scala

@@ -68,7 +74,7 @@ class BarrierTaskContext(
   *
   * CAUTION! In a barrier stage, each task must have the same number of barrier() calls, in all
   * possible code branches. Otherwise, you may get the job hanging or a SparkException after
-   * timeout. Some examples of misuses listed below:
+   * timeout. Some examples of '''misuses''' listed below:


use bold font to make sure users don't misread

nit: listed -> are listed?

just saw it, will include it if Jenkins fails:)

mengxr · 2018-08-27T04:24:30Z

core/src/main/scala/org/apache/spark/BarrierTaskContext.scala

+ */
+@Experimental
+@Since("2.4.0")
+class BarrierTaskContext private[spark] (


Made the constructor package private to force users get it from #get().

SparkQA · 2018-08-27T04:24:52Z

Test build #95276 has finished for PR 22240 at commit 19e380a.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

mengxr · 2018-08-27T04:24:56Z

core/src/main/scala/org/apache/spark/BarrierTaskInfo.scala

@@ -28,4 +28,4 @@ import org.apache.spark.annotation.{Experimental, Since}
 */
 @Experimental
 @Since("2.4.0")
-class BarrierTaskInfo(val address: String)
+class BarrierTaskInfo private[spark] (val address: String)


hide the constructor since this is not to be constructed by user

mengxr · 2018-08-27T04:25:15Z

core/src/main/scala/org/apache/spark/rdd/RDDBarrier.scala

+ */
+@Experimental
+@Since("2.4.0")
+class RDDBarrier[T: ClassTag] private[spark] (rdd: RDD[T]) {


also hide the constructor here

SparkQA · 2018-08-27T06:42:59Z

Test build #95280 has finished for PR 22240 at commit c2624ed.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2018-08-28T07:05:01Z

Test build #95329 has finished for PR 22240 at commit 7e2975b.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

jiangxb1987 · 2018-08-28T16:02:30Z

retest this please

jiangxb1987

LGTM

SparkQA · 2018-08-28T16:21:11Z

Test build #95361 has finished for PR 22240 at commit 7e2975b.

This patch fails to generate documentation.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2018-08-28T23:32:19Z

Test build #95379 has finished for PR 22240 at commit 365e7b8.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

mengxr · 2018-08-29T03:09:45Z

test this please

SparkQA · 2018-08-29T07:05:01Z

Test build #95398 has finished for PR 22240 at commit 365e7b8.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

kiszk · 2018-08-29T12:51:55Z

retest this please

## What changes were proposed in this pull request? I made one pass over the Python APIs for barrier mode and updated them to match the Scala doc in #22240 . Major changes: * export the public classes * expand the docs * add doc for BarrierTaskInfo.addresss cc: jiangxb1987 Closes #22261 from mengxr/SPARK-25248.1. Authored-by: Xiangrui Meng <meng@databricks.com> Signed-off-by: Xiangrui Meng <meng@databricks.com>

SparkQA · 2018-08-29T16:30:36Z

Test build #95418 has finished for PR 22240 at commit 365e7b8.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

jiangxb1987 · 2018-08-29T16:50:04Z

core/src/main/scala/org/apache/spark/scheduler/Task.scala

@@ -82,31 +82,22 @@ private[spark] abstract class Task[T](
    SparkEnv.get.blockManager.registerTask(taskAttemptId)
    // TODO SPARK-24874 Allow create BarrierTaskContext based on partitions, instead of whether
    // the stage is barrier.
-    context = if (isBarrier) {


context is still used in the following statements.

kiszk · 2018-08-31T02:21:03Z

core/src/main/scala/org/apache/spark/BarrierTaskContext.scala

 object BarrierTaskContext {
  /**
+   * :: Experimental ::
   * Return the currently active BarrierTaskContext. This can be called inside of user functions to


nit: Return -> Returns?

## What changes were proposed in this pull request? I made one pass over the Python APIs for barrier mode and updated them to match the Scala doc in apache#22240 . Major changes: * export the public classes * expand the docs * add doc for BarrierTaskInfo.addresss cc: jiangxb1987 Closes apache#22261 from mengxr/SPARK-25248.1. Authored-by: Xiangrui Meng <meng@databricks.com> Signed-off-by: Xiangrui Meng <meng@databricks.com>

SparkQA · 2018-09-03T22:27:42Z

Test build #95639 has finished for PR 22240 at commit b6a3c5b.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2018-09-03T22:32:39Z

Test build #95640 has finished for PR 22240 at commit ae4a8e6.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2018-09-03T22:52:31Z

Test build #95641 has finished for PR 22240 at commit 47ebd08.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2018-09-04T02:38:49Z

Test build #95642 has finished for PR 22240 at commit c61eec3.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2018-09-04T07:05:02Z

Test build #95648 has finished for PR 22240 at commit 9b6a47b.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

jiangxb1987 · 2018-09-04T07:20:25Z

retest this please

SparkQA · 2018-09-04T12:03:27Z

Test build #95661 has finished for PR 22240 at commit 9b6a47b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

jiangxb1987 · 2018-09-04T12:37:56Z

LGTM

mengxr · 2018-09-04T16:56:19Z

Merged into master. Thanks for review!

update barrier Scala doc

19e380a

mengxr commented Aug 27, 2018

View reviewed changes

fix scala style

c2624ed

refactor BarrierTaskContext

7e2975b

mengxr changed the title ~~[WIP] [SPARK-25248] [CORE] Audit barrier APIs for 2.4~~ [SPARK-25248] [CORE] Audit barrier Scala APIs for 2.4 Aug 28, 2018

jiangxb1987 approved these changes Aug 28, 2018

View reviewed changes

make unidoc happy

365e7b8

mengxr mentioned this pull request Aug 29, 2018

[SPARK-25248.1][PYSPARK] update barrier Python API #22261

Closed

jiangxb1987 reviewed Aug 29, 2018

View reviewed changes

kiszk reviewed Aug 31, 2018

View reviewed changes

Merge remote-tracking branch 'apache/master' into SPARK-25248

deb0466

mengxr force-pushed the SPARK-25248 branch from b6a3c5b to ae4a8e6 Compare September 3, 2018 22:12

declare some package private methods to TaskContext

47ebd08

mengxr force-pushed the SPARK-25248 branch from ae4a8e6 to 47ebd08 Compare September 3, 2018 22:36

update mima excludes

c61eec3

mengxr force-pushed the SPARK-25248 branch from 613b4bd to c61eec3 Compare September 3, 2018 22:40

expose getLocalProperties

9b6a47b

asfgit closed this in 061bb01 Sep 4, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-25248] [CORE] Audit barrier Scala APIs for 2.4 #22240

[SPARK-25248] [CORE] Audit barrier Scala APIs for 2.4 #22240

mengxr commented Aug 27, 2018 •

edited

mengxr Aug 27, 2018

mengxr Aug 27, 2018

kiszk Aug 28, 2018

mengxr Aug 29, 2018

mengxr Aug 27, 2018

SparkQA commented Aug 27, 2018

mengxr Aug 27, 2018

mengxr Aug 27, 2018

SparkQA commented Aug 27, 2018

SparkQA commented Aug 28, 2018

jiangxb1987 commented Aug 28, 2018

jiangxb1987 left a comment

SparkQA commented Aug 28, 2018

SparkQA commented Aug 28, 2018

mengxr commented Aug 29, 2018

SparkQA commented Aug 29, 2018

kiszk commented Aug 29, 2018

SparkQA commented Aug 29, 2018

jiangxb1987 Aug 29, 2018

kiszk Aug 31, 2018

SparkQA commented Sep 3, 2018

SparkQA commented Sep 3, 2018

SparkQA commented Sep 3, 2018

SparkQA commented Sep 4, 2018

SparkQA commented Sep 4, 2018

jiangxb1987 commented Sep 4, 2018

SparkQA commented Sep 4, 2018

jiangxb1987 commented Sep 4, 2018

mengxr commented Sep 4, 2018

[SPARK-25248] [CORE] Audit barrier Scala APIs for 2.4 #22240

[SPARK-25248] [CORE] Audit barrier Scala APIs for 2.4 #22240

Conversation

mengxr commented Aug 27, 2018 • edited

What changes were proposed in this pull request?

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SparkQA commented Aug 27, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SparkQA commented Aug 27, 2018

SparkQA commented Aug 28, 2018

jiangxb1987 commented Aug 28, 2018

jiangxb1987 left a comment

Choose a reason for hiding this comment

SparkQA commented Aug 28, 2018

SparkQA commented Aug 28, 2018

mengxr commented Aug 29, 2018

SparkQA commented Aug 29, 2018

kiszk commented Aug 29, 2018

SparkQA commented Aug 29, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SparkQA commented Sep 3, 2018

SparkQA commented Sep 3, 2018

SparkQA commented Sep 3, 2018

SparkQA commented Sep 4, 2018

SparkQA commented Sep 4, 2018

jiangxb1987 commented Sep 4, 2018

SparkQA commented Sep 4, 2018

jiangxb1987 commented Sep 4, 2018

mengxr commented Sep 4, 2018

mengxr commented Aug 27, 2018 •

edited