You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Ah, this would be really interesting! So, Spark recently introduced the concept of a barrier which is meant to be used with deep learning workflows but is generic enough that we can use for anything.
I created a work item a while ago that tracks support for barriers in sparklyr here: #1791
If we could support barrier execution in sparklyr, a user would be free to use Spark executors for whatever they want, including using the future package.
Ah, this would be really interesting! So, Spark recently introduced the concept of a barrier which is meant to be used with deep learning workflows but is generic enough that we can use for anything. ...
@javierluraschi barrier makes spark support MPI mechanism with better failover design. it's awesome for machine leaning. And pyspark pandas udf seems pretty powerful, can we implement it in dplyr way which is a pretty common desired operation?
related to futureverse/future#286, A good idea comes from
future
package.the future package provides a great parallel framework across multi-engine, including socket, fork , mpi, multiprocess and so on.
wish to use future pakcage with spark_apply function.
The text was updated successfully, but these errors were encountered: