Scripts and Examples for Apache Spark on K
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
docs
examples
scripts
README.md
TAG.txt
devel-requirements.txt
requirements.txt
snappy-java.patch.txt
spark.patch.txt

README.md

Using Apache Spark on K

This package provides installation document and utility scripts for running Apache Spark on K Computer. It includes simple examples.

  • The installation document is in docs. Spark (1.6.2) is installed in "/opt/aics/spark/spark-1.6.2-bin-sparkk" on K (on the compute-nodes).

  • The utility scripts are in scripts. The scripts are installed in "/opt/aics/spark/scripts" on K (on the compute-nodes).

  • The examples are in examples. They are provided as job scripts for Fujitsu job manager.

Basics

The scripts are simple wrappers of the scripts of Spark, which help start/stop master and worker processes. Spark runs in the Spark standalone mode, but master/workers are started using MPI (Message Passing Interface) via mpiexec.

The scripts assume that Spark is installed in "/opt/aics/spark" and R in "/opt/aics/R" (on the compute-nodes).

Examples

First try the sample job scripts (for Fujitsu job manager) in examples.

Building Spark

For procedures to build and install, see docs.

More Information

  • For background information using K, but not very specific to Spark, see Basics in docs.