Spark Project

In this project, we try to evaluate the performance of Apache Spark & its MLlib library in function of different parameters: data size, slave nodes, CPU cores...

Data used in the tests comes from a poker hand data set, and you can find it here.

Documentation

• Results this folder contains our graphs and explainations about our test method.
• environment.sh a script you will certainly need to adapt to your Spark installation to set the environment variables required to run PySpark.
• init.sh An init script to download the data, unzip it and set the environment
• performance.py The main script to run
• testTree.py The training method of our program
• training.py called by performance.py to execute the tests and return their results.

Therefore, to run the project:

1. Run ./init.sh
2. Create a Spark master and slave(s) with the script in $SPARK_HOME/sbin
3. Run spark-submit performance.py outputfile NUMBER_OF_PARTITION

note : NUMBER_OF_PARTION define how the RDD will be partitioned inside spark system. Not nough partition and you will end up with not enough parallelisation of the work. For our experiments, we define this number equal to the number of cores available.