Using Apache Spark on K
The installation document is in docs. Spark (1.6.2) is installed in "/opt/aics/spark/spark-1.6.2-bin-sparkk" on K (on the compute-nodes).
The utility scripts are in scripts. The scripts are installed in "/opt/aics/spark/scripts" on K (on the compute-nodes).
The examples are in examples. They are provided as job scripts for Fujitsu job manager.
The scripts are simple wrappers of the scripts of Spark, which help start/stop master and worker processes. Spark runs in the Spark standalone mode, but master/workers are started using MPI (Message Passing Interface) via mpiexec.
The scripts assume that Spark is installed in "/opt/aics/spark" and R in "/opt/aics/R" (on the compute-nodes).
First try the sample job scripts (for Fujitsu job manager) in examples.
For procedures to build and install, see docs.
- For background information using K, but not very specific to Spark, see Basics in docs.