Add port forwardings for Spark and HDFS web apps, e.g. 8080, 4040, ...

[hannes-ucsc]

From @hannes-ucsc on April 18, 2015 7:24

Copied from original issue: BD2KGenomics/cgcloud-spark#2

[hyviquel]

The modification I did by hand to allow this:
Opening Inbound in the security group
Custom TCP 8080-8081 for ALL
Custom Inbound TCP 4040-4045 for ALL
Add export SPARK_PUBLIC_DNS="ec2-xxxxxxxxx.compute.amazonaws.com" to /opt/sparkbox/spark/conf/spark-env.sh

[hannes-ucsc]

@hyviquel, Did you have to do the spark-env.sh modification on every node or just the master. And what wasn't working when you didn't have it?

[hyviquel]

Sorry, I missed your reply.
No, I do not need to modify all spark-env.sh but the Spark cluster has to be restart from the master node using stop-all.sh and start-all.sh scripts.

[hyviquel]

A more generic way to do it is by adding the following code to spark-env.sh :

public_name=$(curl -s http://169.254.169.254/latest/meta-data/public-hostname)
export SPARK_PUBLIC_DNS="$public_name"

[hannes-ucsc]

spark-tools should make the spark-env.sh modification that @hyviquel suggests.