Arrow

[javierluraschi]

Related

• What are the Apache Arrow integration plans? #1327
• Can't pull result back from Spark #1289
• Arrow #1457
• copy_to fails on nested data #1536 (copy_to with nested data)

[mattpollock]

Will this work apply both to spark_apply and to an R version of toPandas? My (limited) understanding is that toPandas uses arrow but that the python version of collect does basically the same thing the R version does.

[javierluraschi]

@mattpollock yes, my intention is to replace all serialization between R and Scala with arrow, with serialization being used in copy_to(), collect() and spark_apply().

[russellpierce]

@javierluraschi Is the expectation then that copy_to() would become suitable for bulk data transfer or would be best pattern still be to get the data off in to a disk storage format for Spark?

[javierluraschi]

@russellpierce copy_to() will never be as efficient as dedicated tools for loading data into a cluster; however, currently, copy_to() has narrow application since it can't really handle even "medium" size data, a few megabytes seems to be the limit. So my hope is that among many improvements introduced by a single data representation with Arrow, we can improve performance and support higher loads. I'm planning to spend this week doing some investigation, should be able to open a WIP PR and share progress and take feedback.

[randomgambit]

@javierluraschi that looks really amazing. will this be shipped with sparklyr 0.9?

[awblocker]

Wanted to check in out timeline for arrow serialization for spark_apply. Glad to help with testing or other open items. My team has a set of workloads (far more groups than executors) where this looks like it will have a very large impact.

[javierluraschi]

@awblocker arrow is already implemented; however, we are waiting for the Arrow 0.12 release to make this a bit more official. For now, you can test this out using the instructions from:

#1611

[skattoor]

After reading the various issues related to this, I'm not quite sure what's the current status : I've installed SparklyR v1.0.1.9004 along with Arrow v0.13.0 and spark_apply still shows the described slowness.

Is there a version I should upgrade to to solve this issue?

Any help much appreciated!