Sparklyr Support Future Frontend.

[harryprince]

related to futureverse/future#286, A good idea comes from future package.

the future package provides a great parallel framework across multi-engine, including socket, fork , mpi, multiprocess and so on.

wish to use future pakcage with spark_apply function.

[javierluraschi]

Ah, this would be really interesting! So, Spark recently introduced the concept of a barrier which is meant to be used with deep learning workflows but is generic enough that we can use for anything.

I created a work item a while ago that tracks support for barriers in sparklyr here: #1791

If we could support barrier execution in sparklyr, a user would be free to use Spark executors for whatever they want, including using the future package.

See also, Barrier Execution Mode.

[harryprince]

Ah, this would be really interesting! So, Spark recently introduced the concept of a barrier which is meant to be used with deep learning workflows but is generic enough that we can use for anything. ...

@javierluraschi barrier makes spark support MPI mechanism with better failover design. it's awesome for machine leaning. And pyspark pandas udf seems pretty powerful, can we implement it in dplyr way which is a pretty common desired operation?

https://databricks.com/blog/2017/10/30/introducing-vectorized-udfs-for-pyspark.html

_{Sent with GitHawk}