[SPARK-5484][GraphX] Checkpoint every 25 iterations in Pregel #4273
Conversation
|
Some users might not want to checkpoint even if their checkpoint directory is set, but I don't a good way to expose that as an option without breaking binary compatibility. @rxin |
|
Test build #26329 has finished for PR 4273 at commit
|
|
You can add a new parameter using a different method name, can't you? |
|
How about adding a new configuration, e.g., spark.graphx.pregel.checkpoint.interval in SparkConf? |
|
And also, this issue seems to be related to SPARK-5561. |
|
Mind tagging this [GRAPHX] so it can get sorted properly? |
ankurdave
changed the title
| @@ -139,6 +141,14 @@ object Pregel extends Logging { | |||
| // get to send messages. We must cache messages so it can be materialized on the next line, | |||
| // allowing us to uncache the previous iteration. | |||
| messages = g.mapReduceTriplets(sendMsg, mergeMsg, Some((newVerts, activeDirection))).cache() | |||
|
|
|||
| if (checkpoint && i % checkpointFrequency == checkpointFrequency - 1) { | |||
There was a problem hiding this comment.
We should output a log if the checkpoint is not active when the checkpoint dir is not set
|
This seems sort of stalled, and https://issues.apache.org/jira/browse/SPARK-5561 suggests this should be accomplished with a more generalized solution anyway. Is it best to close this? |
|
Yeah, let's close this. |
Pregel-based iterative algorithms with more than ~50 iterations begin to slow down and eventually fail with a StackOverflowError due to Spark's lack of support for long lineage chains.
This PR causes Pregel to checkpoint the graph every 25 iterations if the checkpoint directory is set.