Add option to download Spark from a custom URL

[rmessner]

We are trying to setup a cluster with Spark 1.6.1, without HDFS, but it fails.
The reason is that the file downloaded is corrupted, so we can't untar it.

Would be a nice feature to allow user to specify an alternate mirror to download the pre-build spark. This would add a new key to configuration, like :

services:
  spark:
    version: 1.6.0
    preferred-mirror: http://apache.crihan.fr/dist/spark/spark-${spark_version}/${file}

The default value would be https://s3.amazonaws.com/spark-related-packages/${file} and the variables available file ( as the file name of pre-build spark is always the same whatever the mirror is ), spark_version ( e.g. : 1.6.1) , distribution ( e.g. : hadoop2.6 )

Related to #71.

[nchammas]

Thanks for reporting this. I was just thinking about it earlier this week.

The problem you are seeing with Spark 1.6.1 is a known issue upstream that I've been bugging the core Spark maintainers about for a few weeks now. 😞

The Spark 1.6.1 package on S3 that Flintrock uses is corrupt, and unfortunately we don't have the ability to control it. That resource is controlled by the Spark maintainers.

For the record, your proposal here is the Spark analogue to #71, which is for Hadoop.

[rmessner]

My co-workers just told me they have the same issue, so i will make the same fix that i submit on #104 if it's okay for you @nchammas .

[nchammas]

You mean you'll open a PR for #71 that's similar to #104?

[rmessner]

Yes, or make it in the same PR, as you want

[nchammas]

Oh sure, it can be the same PR.

[JoshRosen]

AFAIK the corrupt packages should be fixed now; let me know if they're still a problem.

[nchammas]

Yes, I believe they're fixed now. Thanks for taking care of that @JoshRosen!

Retitling issue accordingly.