A Julia interface to Apache Spark™
Spark.jl is a package that allows the execution of Julia programs on the Apache Spark platform. It supports running pure Julia scripts on Julia data structures, while utilising the data and code distribution capabalities of Apache Spark. It supports multiple cluster types (in client mode), and can be consider as an analogue to PySpark or RSpark within the Julia ecosystem.
Spark.jl requires at least Java 7 and Maven to be installed and available in
This will download and build all Julia and Java dependencies. To use Spark.jl type:
using Spark Spark.init() sc = SparkContext(master="local")
- LATEST — in-development version of the documentation.
The package is tested against Julia
0.6 and Java 7 and 8. It's also been tested on Amazon EMR and Azure HDInsight. While large cluster modes have been primarily tested on Linux, OS X and Windows do work for local development. See the roadmap for current status.
Contributions are very welcome, as are feature requests and suggestions. Please open an issue if you encounter any problems.