Added SparkGCE Script for Version 0.9.1

[sigmoidanalytics]

I have added the SparkGCE script in this pull request. Just like the spark_ec2 script, this one also reads certain command-line arguments (See the README.md for more details) like the cluster name and all, then starts the machines in the google cloud, sets up the network, adds a 500GB empty disk to all machines, generate the ssh keys on master and transfer it to all slaves and install java and downloads and configures Spark-v0.9.1/Shark-v0.9.1/Hadoop-v0.9.1. Also it starts the shark server automatically. Currently the version is 0.9.1 but I'm happy to add/support more versions

[AmplabJenkins]

Can one of the admins verify this patch?

[sigmoidanalytics]

Did any of the admin had chance to check it out? Let me know if you want me to modify anything in it?

[mateiz]

@sigmoidanalytics sorry for the late reply on this, it looks like this got created during the 1.0 release period and reviewers missed it. Hopefully we can get this in 1.2. The main question I have is whether it's possible to share more code between this and the EC2 scripts. Is there any Python library that abstracts over the cloud provider?

[sigmoidanalytics]

Mostly https://libcloud.apache.org/ should work to abstract away some
launch code. Can look more deeply to check stability. Has any body used it
in some other project yet?

Whirr does some cloud level abstraction but would probably not apply here
as we are mostly deploying Spark only using this script.

Regards
Mayur

On Mon, Jul 28, 2014 at 8:00 PM, Matei Zaharia notifications@github.com
wrote:

@sigmoidanalytics https://github.com/sigmoidanalytics sorry for the
late reply on this, it looks like this got created during the 1.0 release
period and reviewers missed it. Hopefully we can get this in 1.2. The main
question I have is whether it's possible to share more code between this
and the EC2 scripts. Is there any Python library that abstracts over the
cloud provider?

—
Reply to this email directly or view it on GitHub
#681 (comment).

Regards,
Mayur Rustagi
Ph: +1 (760) 203 3257
http://www.sigmoidanalytics.com
@mayur_rustagi http://www.twitter.com/mayur_rustagi

[mateiz]

Libcloud looks good actually, and it's nice that it's another Apache project. Would be worth a try if you guys want to investigate it. It would be awesome if we also get OpenStack as a result of it.

[SparkQA]

Can one of the admins verify this patch?

[ash211]

@sigmoidanalytics Did you ever take a look at libcloud to see if we could abstract out some commonality with the spark_ec2 scripts?

[ash211]

Oh and also, all GitHub PRs should have a Jira SPARK-XYZ title in the title. I created one for you -- can you please update this PR name to be "SPARK-4400 Added SparkGCE Script"?

[ash211]

Correction, please use SPARK-1422

[sigmoidanalytics]

LibCloud looks decent for launching machine across datacenters providers,
but to modify build scripts to either do installation from scratch (means
longer install times than spark ec2) or keep slightly baked
AMI's/equivalents in each datacenter of each provider. Both are sizable
amount of effort.
If the purpose of script is for development support then we can choose some
locations & start there before going all out. Another possibility is to
keep this all-datacenter script separate with longer launch times on vanila
AMI for those who want to use other datacenters apart from those supported
by spark-ec2. That way general launch times dont increase & still a
solution for those who dont want to use AWS.

On Fri, Nov 14, 2014 at 4:39 PM, Andrew Ash notifications@github.com
wrote:

Correction, please use SPARK-1422

—
Reply to this email directly or view it on GitHub
#681 (comment).

Regards,
Mayur Rustagi
Ph: +1 (760) 203 3257
http://www.sigmoidanalytics.com
@mayur_rustagi http://www.twitter.com/mayur_rustagi

[pwendell]

This is being maintained in it's own package now, so let's close this issue.