[SPARK-2418] Custom checkpointing with an external function as parameter

[Forevian]

https://issues.apache.org/jira/browse/SPARK-2418

If a job consists of many shuffle heavy transformations the current resilience model might be unsatisfactory. In our current use-case we need a persistent checkpoint that we can use to save our RDDs on disk in a custom location and load it back even if the driver dies. (Possible other use cases: store the checkpointed data in various formats: SequenceFile, csv, Parquet file, MySQL etc.)
After talking to Patrick Wendell at the Spark Summit 2014 we concluded that a checkpoint where one can customize the saving and RDD reloading behavior can be a good solution. I am open to further suggestions if you have better ideas about how to make checkpointing more flexible.

---

Note1: I deleted the previous fork as I had messed that version up by some unsuccessful rebasing attempt.
Note2: The contribution is my original work and I license the work to the project under the project's open source license.

[AmplabJenkins]

Can one of the admins verify this patch?

[Forevian]

@pwendell, I have adopted it for the recent spark core master

[AmplabJenkins]

Can one of the admins verify this patch?

[AmplabJenkins]

Can one of the admins verify this patch?

[andrewor14]

@Forevian thanks for submitting the patch. Unfortunately it has mostly gone stale at this point and many changes have gone into master between now and when it was created. Since it's unlikely to be merged, would you mind closing this PR? Feel free to reopen it against the same issue and we can start the discussion there.

[darabos]

Thanks for the note, @andrewor14. @Forevian is not working with Spark lately, but I'm happy to take over this change from him. From a superficial look at the code it seems to me that the same approach would still work. It's a fantastically flexible solution yet it's entirely backward-compatible. Since the old code path would remain unchanged, there is also very little risk in it.

So I'd like to dust it off and send a new pull request against the current master. I'd just like to ask first if you have any recommendations for avoiding the same fate as this pull request. Why was it never reviewed? Why did it not get any comments? Thanks!

[andrewor14]

Hi @darabos, in my experience I have noticed that PRs that do get attention are usually high priority issues or patches that many in the community are interested in. Pinging committers for review is a start, but it also really helps if the original author is active and responsive .

In this particular case, @pwendell has been busy with many other things and hasn't really had time to do reviews for many patches, so maybe he is not the best person to ping. For core patches like this, you could ping me and I will either review it myself or try to triage them to the right person.