Load some recipes from a file, select the recipes with chilies (singular, plural and with typo) and export them into a parquet file. This was a technical challenge to skill myself with Spark.
- Install python 2.7.14, like using pyenv:
pyenv install 2.7.14 - Install requirements, like with pip
pip install -r requirements.txt - Install Spark 1.6.1
- Add package pySpark
- Ensure NLTK dependencies are up to date:
python -c "import nltk;nltk.download()" - Download input dataset in the project folder, like with wget:
wget https://s3-eu-west-1.amazonaws.com/dwh-test-resources/recipes.json .
Local mode
python spark.py local
In Standalone
- Start services
sbin/start-master.sh -h 127.0.0.1 # start spark master
sbin/start-slave.sh spark://127.0.0.1:7070 # add exeutor to master
- Launch application with
spark-submit. For instance, on my local installation:
~/sys/spark-1.6.1-bin-hadoop2.4/bin/spark-submit spark.py spark://localhost:7077
Over YARN
~/sys/spark-1.6.1-bin-hadoop2.4/bin/spark-submit spark.py yarn-client
On top of each commit, all tests must pass:
py.test && rm -r **/*.pyc **/__pycache__
On top of each commit, no offense must be detected:
pep8 -- .