SPARK-3874, Provide stable TaskContext API

[ScrapCodes]

No description provided.

[AmplabJenkins]

Can one of the admins verify this patch?

[SparkQA]

QA tests have started for PR 2782 at commit ef633f5.

• This patch merges cleanly.

[SparkQA]

QA tests have finished for PR 2782 at commit ef633f5.

• This patch fails to build.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • public abstract class TaskContext implements Serializable

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21687/
Test FAILed.

[ScrapCodes]

Jenkins, retest this please.

[SparkQA]

QA tests have started for PR 2782 at commit ef633f5.

• This patch merges cleanly.

[SparkQA]

QA tests have finished for PR 2782 at commit ef633f5.

• This patch fails to build.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • public abstract class TaskContext implements Serializable

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21688/
Test FAILed.

[SparkQA]

QA tests have started for PR 2782 at commit bbd9e05.

• This patch merges cleanly.

[SparkQA]

QA tests have finished for PR 2782 at commit bbd9e05.

• This patch fails MiMa tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • public abstract class TaskContext implements Serializable

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21689/
Test FAILed.

[JoshRosen]

Is there any easy way to reduce the visibility of markTaskCompleted() and the other internal methods? Could we maybe move those to TaskContextImpl and leave TaskContext as an abstract base class with no actual methods actually implemented (if I recall, we can't convert it to an interface for binary compatibility reasons)?

If we do that, you could use some tricks to cut down on the amount of boilerplate getter code code in TaskContext. In #2696, for example, I use this pattern:

Java interface / abstract base class:

public interface SparkJobInfo {
  int jobId();
  int[] stageIds();
  String status();
}

Scala implementation:

private class SparkJobInfoImpl (
  val jobId: Int,
  val stageIds: Array[Int],
  val status: String)
 extends SparkJobInfo

[pwendell]

I'd prefer this class not have a constructor (the whole idea here is that we don't want user to instantiate it). Basically we want to move all of the logic here to TaskContextImpl and only define the interface here in terms of abstract methods.

[pwendell]

Hey @ScrapCodes - this is a good start, but I'd prefer to move all of the functionality into TaskContextImpl. Basically, we want TaskContext to look like an interface definition and make it so users can't construct one directly. The only reason we aren't making this a proper interface is because of compatibility reasons.

[pwendell]

By the way - if you look at the original design I put in the JIRA, this is what it does. Thinks like markTaskCompleted should be in the Impl class.

https://issues.apache.org/jira/browse/SPARK-3874

[ScrapCodes]

Hey Patrick, You are right about that.

We can make TaskContext an interface if we only allow TaskContextHelper.get() instead of TaskContext.get(). And then maybe I can rename TaskContextHelper to TaskContextGetter ? (Not sure !)

[pwendell]

@ScrapCodes We cant make it a proper interface because of binary compatibility (this is a pretty widely used API so I'd prefer not to break it).

[pwendell]

//cc @rxin - should this be "isComplete" rather than "isCompleted"?

[pwendell]

Actually this looks good - we use isCompleted elsewhere already in user-facing code.

[rxin]

My understanding is that isCompleted means it has been completed, whereas isComplete means it is "whole".

[JoshRosen]

Yeah, I think isCompleted is right, since we'd use isFailed instead of isFailure or isFail.

[pwendell]

Jenkins, test this please

[SparkQA]

QA tests have started for PR 2782 at commit ed551ce.

• This patch merges cleanly.

[SparkQA]

QA tests have finished for PR 2782 at commit ed551ce.

• This patch fails to build.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • public abstract class TaskContext implements Serializable

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21729/
Test FAILed.