[SPARK-7467] Dag visualization: treat checkpoint as an RDD operation

[andrewor14]

Such that a checkpoint RDD does not go into random scopes on the UI, e.g. take. We've seen this in streaming.

[AmplabJenkins]

Merged build triggered.

[andrewor14]

@zsxwing @tdas

[AmplabJenkins]

Merged build started.

[SparkQA]

Test build #32217 has started for PR 6004 at commit 19bc07b.

[SparkQA]

Test build #32217 has finished for PR 6004 at commit 19bc07b.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Merged build finished. Test PASSed.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32217/
Test PASSed.

[zsxwing]

I just tried this PR. But looks it does not work.

Test codes:

    sc.setCheckpointDir(".")
    val r1 = sc.parallelize(1 to 100)
    r1.checkpoint()
    r1.map(_ + 2).filter(_ < 50).collect()
    r1.map(_ + 2).filter(_ < 50).collect()

I opened the spark shell and typed the above codes. And the dag of job 2 is

The first one is still collect.

[andrewor14]

Oops, thanks for the screen shot. It appears that it's still not being captured correctly.

[andrewor14]

By the way I have a simpler reproduction:

sc.setCheckpointDir("/tmp")
val rdd = sc.parallelize(1 to 10)
rdd.checkpoint()
rdd.take(10)

Since the take runs multiple jobs and we do the actual checkpointing at the end of each job.

[andrewor14]

I think I fixed it in the latest commit.

[AmplabJenkins]

Merged build triggered.

[AmplabJenkins]

Merged build started.

[SparkQA]

Test build #32296 has started for PR 6004 at commit 9217439.

[SparkQA]

Test build #32296 has finished for PR 6004 at commit 9217439.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class HasCheckpointInterval(Params):
  • class ALS(JavaEstimator, HasCheckpointInterval, HasMaxIter, HasPredictionCol, HasRegParam, HasSeed):
  • class ALSModel(JavaModel):

[AmplabJenkins]

Merged build finished. Test FAILed.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32296/
Test FAILed.

[andrewor14]

retest this please

[AmplabJenkins]

Merged build triggered.

[AmplabJenkins]

Merged build started.

[SparkQA]

Test build #32308 has started for PR 6004 at commit 9217439.

[SparkQA]

Test build #32308 has finished for PR 6004 at commit 9217439.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Merged build finished. Test PASSed.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32308/
Test PASSed.

[andrewor14]

Ok I'm merging this into master 1.4.