Formula to setup and configure specific versions of Apache Spark.
Note
See the full Salt Formulas installation and usage instructions.
Install Apache Spark
spark.worker ------------Installs and sets up a stand-alone spark worker/slave server
spark.master ------------Installs and sets up a stand-alone spark master server.
Helpful for debugging, dumps the jinja map to a text file in /tmp
If all you need is the Spark environment, not a worker or master, add spark or spark.env to the topfile; Otherwise, add spark.master or spark.worker (or both!) to the appropriate include or topfile declaration.
'G@roles:spark-worker':
- spark.worker
'G@roles:spark-master':
- spark.masterOverriding the pillar is Testing =========
Testing is done with KitchenSalt which means you'll also need a working Ruby setup and preferably at least 2.2, but you can use whatever version as long as you update the Gemfile. You will also need bundler installed and can be done so with gem install bundler.
If all that works, you should be able to run kitchen test which is an alias for kitchen converge + kitchen verify but it deletes the box on completion so it isn't very useful during development.
# Initial setup
which bundle || gem install bundler
bundle install
# build vagrant box and run states
kitchen converge
# run tests in `test/integration/default`
kitchen verify
# sledgehammer
kitchen destroy
# alias for running (destroy + converge + verify + destroy)
kitchen testworker.sls and master.sls are identical, maybe it's time to combine them into a service.sls and interrogate salt['pillar.get']('grains', []) using spark:lookup:worker_service and spark:lookup:master_service to determine which service the state declarations should operate on.