[SPARK-4948] Use pssh. #85
Conversation
|
cc @shivaram |
|
LGTM. Thanks for testing this through. I'd like to do one last test by launching a cluster using this branch before merging -- Will do it by tomm. |
|
So that we don't accidentally break things for users that are running a released Spark 1.2.0 version, do you think that we should merge this into a new |
|
Though this should be a seamless change, yes it would be safer to merge this into a new branch. Should we take this opportunity to start naming the branches after major Spark releases? So instead of I think it will make this stuff simpler to manage going forward. |
|
That naming convention sounds fine to me; I just bumped the |
|
What dost the @shivaram thinketh? |
|
If we are going to create new branches lets call them the same names as the Spark versions going forward. I wasn't very convinced the last time around that we needed a branch per version -- though I now see that it is just better to be more careful and ensure we don't break anything for released versions. I'd be fine with starting a v1.3 branch with this PR. @pwendell -- Any thoughts on this ? P.S: Given that we are changing conventions, it would be good to document somewhere how v1, v2, v3, v4 map to Spark versions. |
|
Taking a quick look, these are the mappings I see.
The reason for the two branch columns is that until this commit, we specified the branch in two separate locations: once to get the AMI list, and once to get the script files to install Hadoop, Ganglia, etc. Up to version 0.7.2, AMI IDs were downloaded from S3 and |
|
Thanks @nchammas -- This is great ! I'd suggest putting this in the Spark wiki, so we have this information around for later. Also while the two columns make sense, it would be good to note below the table that the ami-list did not change between v2 and v3 (hence the mapping is consistent). I don't think we will back port AMI changes to v3, so this should remain true going forward. |
|
One last minor nit: The spark-ec2 repo was only used beginning in 0.7 [1], before that we just had an AMI that was pre-baked with the scripts. So anything before 0.7 isn't really supported using this repo. [1] This commit introduced the spark-ec2 repo apache/spark@d012cfa |
I don't have write access to the wiki, I don't think. |
|
I just gave the wiki user id nchammas permission to add pages. Let me know if it doesn't work |
|
I'll fill it in later this week. |
|
Wiki looks good - I'll create a new branch and we can move the PR to that branch. BTW any suggestions on naming ? I was going to use |
|
@pwendell hasn't had a chance to chime in yet, but |
|
Yeah that branch naming sounds good to me. It makes more sense. On Wed, Dec 24, 2014 at 11:01 AM, Nicholas Chammas <notifications@github.com
|
|
@nchammas I just pushed out a new branch to mesos/spark-ec2 named branch-1.3 https://github.com/mesos/spark-ec2/tree/branch-1.3 -- Can you move this PR to that branch ? |
|
Done: #86 |
|
@shivaram Do you also need to make |
|
Yeah I dont have permissions to do that though. @pwendell @JoshRosen can one of you make branch-1.3 the default branch for mesos/spark-ec2 ? |
|
I don't have permission either; we need someone with repo admin access to do it (I think @pwendell can). |
Going forward, we'll use matching branch names across the mesos/spark-ec2 and apache/spark repositories, per [the discussion here](mesos/spark-ec2#85 (comment)). Author: Nicholas Chammas <nicholas.chammas@gmail.com> Closes #3804 from nchammas/patch-2 and squashes the following commits: cd2c0d4 [Nicholas Chammas] [EC2] Update mesos/spark-ec2 branch to branch-1.3
|
Pinging @pwendell about making |
|
Done, thanks guys. On Tue, Jan 13, 2015 at 6:28 PM, Nicholas Chammas notifications@github.com
|
psshto neatly parallelize cluster operations.