Request for *_ply to take parallel?

[jameshowison]

Sorry if this is the wrong place, but I wondered why d_ply doesn't have a parallel argument, I'm splitting up a data frame and drawing graphs with it, seems a good use for parallelization ... did a quick search of the list and Hadley says it isn't supported back in Feb. Is this something that might be supported?

Thanks,
James

[hadley]

Yes, definitely. Just need to find the time to work on it.

[myersdb]

Glad to see this will be happening. I commonly operate on a large set of large files, performing batch operations of GIS analysis via sp and raster on a large cluster. Similar to the above case where intermediate and final products are stored as files rather than as R objects.

[kenahoo]

While we're on this topic - would it be beneficial for the **ply() family of functions to also take an argument indicating whether side-effects should be propagated or not? If they knew that they only needed to transmit the return value back to the caller, and not transmit the entire set of changes that happened to the R environment, they could optimize the parallel execution environment.

Or is that best handled by encapsulating the behavior in the foreach backend?

[hadley]

I think that should be the default - and if you want side effects propagated outside, you need to use the parallel tools directly.

[kenahoo]

Yeah, that seems reasonable, because it's not clear what automatic/default strategy should be used to merge all the effects back in to the main process.