This example contains code for running DL4J on Spark standalone as well as normal Spark.
DL4J interoperates with Spark via dedicated classes and workers that take a Spark context and a neural net configuration.
Underneath the hood, we create the associated RDDs and other things that allow us to perform distributed neural-net training on a cluster.
We scale out via Hogwild! and iterative reduce parameter averaging.
#ND4J on Spark
To run Spark with different Blas backends, just include the backend dependency you want in your project's POM.xml file to make sure it gets on the classpath. When these JAR files are included on the classpath, those will be included as part of the JAR file for spark-submit. You can learn more in Spark's documentation.
If you run mvn clean package on Spark for a Spark job, the backend dependencies you include in the POM.xml will already be included.
You can create a JAR file appropriate for spark-submit via the Apache Maven Shade plugin (included in this pom). You can also run Spark standalone. We have one example for each.
#Spark.ml
If you would like to use the newer spark.ml, please see this page.